Visual presentation of gingival line generated based on 3d tooth model

ABSTRACT

Systems and methods of simulating dental treatments are disclosed. A method may include capturing a first 2D image of a patient&#39;s face, including the patient&#39;s teeth, building a parametric 3D model of the patient&#39;s teeth and gingiva based on the first 2D image, developing a simulated outcome of a dental treatment of the patient&#39;s teeth by rendering the parametric 3D model with the patient&#39;s teeth in one or more positions and/or orientations corresponding to the treatment goals of the dental treatment plan, and rendering a second 2D image of the patient&#39;s face with gingiva and teeth according to a simulated outcome of the dental treatment plan. As noted herein, the dental treatment plan may include orthodontic and/or restorative elements. The simulated outcome may correspond to estimated outcomes and/or intended outcomes of the dental treatment plan.

CROSS-REFERENCE

This application claims the benefit of U.S. Provisional Application No.62/847,780, filed May 14, 2019, which application is incorporated hereinby reference.

TECHNICAL FIELD

The technical field relates to digital dental technologies, and moreparticularly to providing a simulated outcome of dental (e.g.,orthodontic, restorative, etc.) treatment by evaluating atwo-dimensional (2D) depiction of a patient's untreated teeth againstparameters associated with model tooth arches.

BACKGROUND

Orthodontic treatment often includes addressing issues withmalpositioned teeth and/or jaws and may include diagnosis, prevention,and/or correction of malocclusions. A person seeking orthodontictreatment may seek a treatment plan from an orthodontist, such as aprofessional who has undergone special training after graduating fromdental school. Many orthodontic treatment plans include treatment withbraces, brackets, wires, and/or polymeric appliances. A person seekingorthodontic treatment may have orthodontic appliances adjusted atvarious times by an orthodontic professional who designs and/orimplements the orthodontic appliances.

Many people are referred for orthodontic treatment by dentists, othertreatment professionals, or other people. For instance, many teenagepatients or people with severe malocclusions may be referred fororthodontic treatment by their dentists or parents. However, many otherpeople may not know whether or not they should receive orthodontictreatment. As an example, many individuals with mild malocclusions maynot know whether or not orthodontic treatment is appropriate ordesirable for them.

Additionally, many people may wish to visualize examples of their smileswithout malpositioned teeth and/or jaws, e.g., after estimated and/orintended dental treatment(s), after insertion of implants or otherdevices, with configurations that would be appropriate for their face,age, heritage, and/or lifestyle, etc. While it may be desirable toprovide people with the ability to visualize how their smiles and/orfaces would look after possible treatment options, the computationalburdens and/or costs of existing tools make it hard to do so. Existingtools also make it hard to visualize how dental treatment wouldmeaningfully impact patients' lives.

SUMMARY

This disclosure generally relates to systems, methods, and/orcomputer-readable media related to simulating dental treatment of apatient's teeth, and particularly to providing a more photo-realisticrendering of a two-dimensional (2D) image of a patient that representsone or more simulated (e.g., estimated and/or intended) outcomes of adental treatment plan. The implementations herein produce near accurateand realistic renderings of simulated outcomes of dental treatmentand/or animations of three-dimensional (3D) models that could previouslyhave not been generated, or generated only in a rudimentary fashionthrough manual photo editing tools. As noted herein, the implementationsdescribed use automated agents and/or rules to provide accurate andrealistic renderings of simulated outcomes of dental (e.g., orthodontic,restorative, etc.) treatment and/or animations of 3D models that werepreviously not possible. The implementations herein allow peopleconsidering and/or undergoing orthodontic treatment the ability tovisualize on a computer automatically generated simulated orthodontictreatment outcome simulations, and may inform a person's choices as towhether or not to seek orthodontic treatment in general and/or specificcourses of orthodontic treatment. As noted herein, the disclosure alsorelates to systems and methods of accurately and realisticallysimulating 3D models of teeth in final orthodontic positions in a 2Dimage of person.

A computer-implemented method of simulating one or more simulatedoutcomes of dental treatment is disclosed. In some embodiments, thecomputer-implemented method of simulating orthodontic treatment mayinclude capturing a first 2D image. In some embodiments, the first 2Dimage may include a representation of a patient's face and a patient'steeth. The method may include identifying one or more shapes associatedwith at least one of the patient's teeth. The method may also includebuilding a parametric 3D model of the patient's teeth based on the first2D image, using one or more case-specific parameters for the one or moreshapes associated with the at least one of the patient's teeth. Themethod may also include simulating an outcome of a dental treatment planfor the patient's teeth to produce a simulated outcome of the dentaltreatment plan; and modifying the parametric 3D model to provide amodified 3D model representing the simulated outcome of the dentaltreatment plan. The method may also include rendering, using themodified 3D model, a second 2D image representing the patient's face,wherein the second 2D image represents the patient's teeth in accordancewith the simulated outcome of the dental treatment plan.

In some embodiments, building the parametric 3D model includes findingedges of teeth and lips in the first 2D image, aligning a parametrictooth model to the edges of the teeth and lips in the first 2D image todetermine the case-specific parameters, and storing the case-specificparameters of the parametric tooth model that align the parametric toothmodel with the edges of the teeth, gingiva, and lips in the first 2Dimage.

In some embodiments, rendering the second 2D image includes accessingthe parametric 3D model of the patient's teeth, projecting one or moreteeth positions from the first 2D image onto the parametric 3D model,and mapping color data from the 2D image to corresponding locations onthe parametric 3D model to generate textures for the parametric 3Dmodel, and using the textures as part of the second 2D image of thepatient's face.

In some embodiments, the predetermined position is based on an averageposition of a plurality of previous patients' teeth after dentaltreatment.

In some embodiments, the predetermined position is based on an averageposition of a plurality of previous patients' teeth before dentaltreatment.

In some embodiments, the computer-implemented method may include findingedges of teeth and lips in the first 2D image, and aligning theparametric 3D tooth model to the edges of the teeth, gingiva, and lipsin the first 2D image.

In some embodiments, the first 2D image may include a profile imagerepresenting a profile of the patient's face.

In some embodiments, the simulated outcome of the dental treatment planmay include an estimated outcome of the dental treatment plan.

In some embodiments, the simulated outcome of the dental treatment planmay include an intended outcome of the dental treatment plan.

In some embodiments, the dental treatment plan may include anorthodontic treatment plan, a restorative treatment plan, or somecombination thereof.

In some embodiments, capturing the first 2D image may includeinstructing a mobile phone or a camera to image the patient's face, orgathering the first 2D image from a storage device or a networkedsystem.

In some embodiments, building the parametric 3D model of the patient'steeth based on the 2D image, using one or more case-specific parametersfor the one or more shapes associated with the at least one of thepatient's teeth may include coarsely aligning teeth represented in the3D parametric model to the patient's teeth represented in the 2D image,and executing an expectation step to determine a probability that aprojection of a silhouette of the 3D parametric model matches one ormore edges of the 2D image a first time.

In some embodiments, building the parametric 3D model of the patient'steeth based on the 2D image, using one or more case-specific parametersfor the one or more shapes associated with the at least one of thepatient's teeth may include executing a maximization step using a smallangle approximation to linearize the rigid transformation of the teethin the 3D model, and executing the expectation step to determine aprobability that a projection of a silhouette of the 3D parametric modelmatches the edges of the 2D image a second time.

In some embodiments, the computer-implemented method may includeiterating through the expectation and maximization steps a firstplurality of times to with a first subset of parameters, and afteriterating through the expectation and maximization steps the firstplurality of times with the first subset of parameters of the 3Dparametric model, iterating though the expectation and maximizationsteps the second plurality of times with the first and second subset ofparameters.

In some embodiments, the computer-implement method may include capturinga first 2D image of a patient's face, including their teeth. The methodmay include building a parametric 3D model of the patient's teeth basedon the 2D image, the parametric 3D model including case-specificparameters for the shape of at least one of the patient's teeth. Themethod may also include simulating an estimated (e.g., an estimatedfinal) and/or intended (e.g., an intended final) orthodontic position ofthe patient's teeth by gathering information about one or more modeltooth arches that represent smiles without malpositioned teeth and/orjaws, and by rendering the 3D model with the patient's teeth in apredetermined position (e.g., one corresponding to positions of teeth inthe model arches) and rendering a second 2D image of the patient's facewith teeth in an estimated orthodontic position.

In some embodiments, building a parametric 3D model includes findingedges of teeth and lips in the first 2D image, aligning a parametrictooth model to the edges of the teeth and lips in the first 2D image todetermine the case-specific parameters, and storing the case-specificparameters of the parametric tooth model that align the parametric toothmodel with the edges of the teeth, gingiva, and lips in the first 2Dimage.

In some embodiments, rendering the second 2D image includes renderingthe parametric model of the patient's according to the position of theteeth in the first 2D image, projecting the 2D image onto the renderedparametric model of the patient's according to the position of the teethin the first 2D image, and mapping the color data from the 2D image tocorresponding locations on the 3D model to generate textures for the 3Dmodel, and rendering the second 2D image of the patient's face withteeth in the estimated orthodontic position with the generated textures.

In some embodiments, rendering the second 2D image further includesapplying simulated treatments or viewing customizations to the second 2Dimage.

In some embodiments, the simulated treatments or viewing customizationsmay include one or more of changing an edge of the gingiva, replacing atooth, adjusting a jaw position, or adjusting the color data.

In some embodiments, the predetermined position is based on acombination of (e.g., an average) positions of a plurality of previouspatients' teeth after orthodontic treatment and/or without misalignedteeth or jaws.

In some embodiments, the predetermined position is based on acombination of (e.g., an average) positions of a plurality of previouspatients' teeth before orthodontic treatment.

In some embodiments, the method may include finding edges of teeth andlips in the first 2D image, and aligning the parametric tooth model tothe edges of the teeth, gingiva, and lips in the first 2D image.

In some embodiments, the first 2D image comprises a profile image.

A computer-implemented method of building a 3D model of teeth from a 2Dimage is disclosed. The method may include capturing a 2D image of apatient's face, including their teeth, determining edges of teeth andgingiva within the first 2D image, fitting the teeth in a 3D parametricmodel of teeth to the edges of the teeth and gingiva within the first 2Dimage, the 3D parametric model including case-specific parameters forthe shape of the patient's teeth, determining the value of thecase-specific parameters of the 3D parametric model based on thefitting.

In some embodiments, the fitting of the teeth in the 3D parametric modelof teeth to the edges of the teeth and gingiva within the first 2D imagecomprises: coarsely aligning the teeth in the 3D parametric model to theteeth in the 2D image, and executing an expectation step to determine aprobability that a projection of a silhouette of the 3D parametric modelmatches the edges of the 2D image.

In some embodiments, the fitting of the teeth in the 3D parametric modelof teeth to the edges of the teeth and gingiva within the first 2D imagefurther comprises: executing a maximization step using a small angleapproximation to linearize the rigid transformation of the teeth in themodel, and executing the expectation step to determine a probabilitythat a projection of a silhouette of the 3D parametric model matches theedges of the 2D image again.

In some embodiments, a computer-implemented method further comprisesiterating through the expectation and maximization steps a firstplurality of times with a first subset of parameters; and afteriterating through the expectation and maximization steps the firstplurality of times with the first subset of parameters of the 3Dparametric model, iterating through the expectation and maximizationsteps the second plurality of times with the first and second subset ofparameters.

In some embodiments, the first plurality of times is the same as thesecond plurality of times.

In some embodiments, the first subset of case-specific parameters of the3D parametric model is one or more of a scale factor, tooth location,and orientation.

In some embodiments, the second subset of parameters of the 3Dparametric model are one or more of a tooth shape and tooth location andorientation.

A computer-implemented method of providing a simulated outcome oforthodontic treatment is disclosed. The method may include building a 3Dparametric model of an arch, the 3D parametric model comprising genericparameters for tooth shape, tooth position, and tooth orientation,capturing a 2D image of a patient, constructing a case-specific 3Dparametric model of the patient's teeth from the 2D image, determiningthe case-specific parameters of the constructed parametric model,rendering the 3D parametric model of the patient's teeth in an estimatedand/or intended final position (e.g., without misaligned teeth and/orjaws), and inserting the rendered 3D model into a 2D image of thepatient.

In some embodiments, the constructing of a 3D parametric model of thepatient's teeth from the 2D image includes: finding edges of teeth,gingiva, and lips in the first 2D image, and aligning the 3D modelparametric to the edges of the teeth, gingiva, and lips in the first 2Dimage.

In some embodiments, the method further comprises applying textures tothe rendered 3D parametric model of the patient's teeth in an estimatedand/or intended final position (e.g., without misaligned teeth and/orjaws), wherein the textures are derived from the 2D image of thepatient.

In some embodiments, the textures are derived from the 2D image of thepatient by: projecting the 2D image onto the rendered parametric modelof the patient's according to the position of the teeth in the first 2Dimage, and mapping the color data from the 2D image to correspondinglocations on the 3D model to derive the textures for the 3D model.

In some embodiments, the rendering of the 3D parametric model of thepatient's teeth in the estimated and/or intended final positioncomprises: generating a mean shape of teeth based on the 3D parametricmodel, adjusting the shape of the teeth in the 3D parametric model basedon case-specific tooth shape parameters, positioning the teeth in a meantooth location and orientation based on a mean location and orientationparameter such that the teeth have a case specific shape and a meanlocation and orientation, and scaling the arch based on a case-specificarch scaling parameter.

A non-transitory computer readable medium includes instruction that whenexecuted by a processor cause the processor to perform any of themethods described herein.

A system is disclosed. The system may include a photo parameterizationengine configured to generate a 3D parametric arch model from a 2D imageof a patient's face and teeth, the parametric 3D model includingcase-specific parameters for the shape of at least one of the patient'steeth and a parametric treatment prediction engine configured toidentify an estimated and/or intended outcome of orthodontic treatmentof a patient based on the 3D parametric arch model and historic modelsof a plurality of patients and/or idealized tooth arch models.

In some embodiments, the system includes a treatment projectionrendering engine configured to render the 3D parametric arch model.

In some embodiments, the photo parameterization engine, the parametrictreatment prediction engine, and the treatment projection renderingengine are together configured to perform the methods described herein.

INCORPORATION BY REFERENCE

All publications, patents, and patent applications mentioned in thisspecification are herein incorporated by reference to the same extent asif each individual publication, patent, or patent application wasspecifically and individually indicated to be incorporated by reference.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel features of the invention are set forth with particularity inthe appended claims. A better understanding of the features andadvantages of the present invention will be obtained by reference to thefollowing detailed description that sets forth illustrative embodiments,in which the principles of the invention are utilized, and theaccompanying drawings of which:

FIG. 1 illustrates a method of providing an estimated outcome oforthodontic treatment, in accordance with one or more embodimentsherein;

FIG. 2 illustrates a parametric tooth model, in accordance with one ormore embodiments herein;

FIG. 3A illustrates an example of how well a parametric tooth modelmatches an original 3D model, in accordance with one or more embodimentsherein;

FIG. 3B illustrates a method of determining generic parameters fromhistoric and/or idealized cases, in accordance with one or moreembodiments herein;

FIG. 4 illustrates the alignment of past cases for use in determiningparameters of a parametric model, in accordance with one or moreembodiments herein;

FIG. 5 depicts a method of generating a parametric model of a patient'steeth, and converting the parametric model into a 3D model of a dentalarch, in accordance with one or more embodiments herein;

FIG. 6 depicts a method for constructing a 3D model from a 2D image, inaccordance with one or more embodiments herein;

FIG. 7A depicts a method of building a patient-specific parametric modelof a patient's teeth, in accordance with one or more embodiments herein;

FIG. 7B depicts tooth models with gingiva and lip edges, in accordancewith one or more embodiments herein;

FIG. 8 depicts a method of rendering patient's teeth in an initialposition using a parametric model of the patient's arch, in accordancewith one or more embodiments herein;

FIG. 9 depicts a method of constructing and applying textures to a 3Dmodel, in accordance with one or more embodiments herein;

FIG. 10A depicts a method of simulating an estimated outcome of anorthodontic treatment on a patient's teeth, in accordance with one ormore embodiments herein;

FIG. 10B depicts a method of simulating orthodontic treatment of apatient, based on matching tooth shape parameters, in accordance withone or more embodiments herein;

FIG. 11 shows an example of a method for rendering teeth according to anestimated outcome of a dental treatment plan, in accordance with one ormore embodiments herein;

FIG. 12 depicts a system for simulating an estimated outcome of anorthodontic treatment, in accordance with one or more embodimentsherein;

FIG. 13 depicts an example of one or more of the elements of theestimated orthodontic treatment simulation system, in accordance withone or more embodiments herein;

FIG. 14 illustrates a tooth repositioning appliance, in accordance withone or more embodiments herein;

FIG. 15 illustrates a tooth repositioning system, in accordance with oneor more embodiments herein;

FIG. 16 illustrates a method of orthodontic treatment using a pluralityof appliances, in accordance with one or more embodiments herein;

FIG. 17 illustrates a method for designing an orthodontic appliance, inaccordance with one or more embodiments herein;

FIG. 18 illustrates a method for planning an orthodontic treatment, inaccordance with one or more embodiments herein;

FIG. 19 is a simplified block diagram of a system for designing anorthodontic appliance and planning an orthodontic treatment, inaccordance with one or more embodiments herein;

FIG. 20 illustrates a method of simulating a gingiva line, in accordancewith one or more embodiments herein;

FIG. 21A illustrates gingiva line input determined from a 2D image of apatient's teeth, in accordance with one or more embodiments herein;

FIG. 21B depicts a process of modeling a gingiva line based on inputdetermined from a 2D image of a patient's teeth, in accordance with oneor more embodiments herein;

FIG. 22 illustrates gingiva line parameters, in accordance with one ormore embodiments herein;

FIG. 23 illustrates a method of leveling a gingiva line, in accordancewith one or more embodiments herein;

FIG. 24 depicts a process of rendering a realistic composite image ofpatient's face and a model of a patient's teeth, in accordance with oneor more embodiments herein.

DETAILED DESCRIPTION

The implementations discussed herein provide tools, such as automatedagents, to visualize the effect of correcting malpositioned teeth/jaws,malocclusion, etc., without the computational burden(s) and/orexpense(s) of scanning a patient's dentition or impressions of thedentition, and to calculate final positions of a treatment plan on apatient's dentition, etc. As discussed in detail herein, thesetechniques may involve obtaining a two-dimensional (2D) representation(such as an image) of a patient's dentition, obtaining one or moreparameters to represent attributes of the patient's dentition in the 2Drepresentation, and using the one or more parameters to compare theattributes of the patient's dentition with attributes of model arches,such as those of historical cases and/or those representing idealizedarch forms. The techniques herein may provide a basis to simulate asimulated outcome of a dental treatment plan.

A “simulated outcome of a dental treatment plan,” as used herein, mayinclude an estimated and/or intended outcome of the dental treatmentplan, such as after implementation of one or more dental procedures,such as orthodontic procedures, restorative procedures, etc. An“estimated outcome of a dental treatment plan,” as used herein, mayinclude an estimate of a state of a patient's dentition after dentalprocedures. An estimated outcome of a dental treatment plan, as usedherein, may, in some instances, be different from an “actual outcome ofa dental treatment plan,” which may represent the state of a patient'sdentition after implementation of the dental treatment plan. Estimatedand actual outcomes of a dental treatment plan, as used herein, may, invarious instances, be different from an “intended outcome of a dentaltreatment plan,” which may represent the intended state of a patient'sdentition after implementation of a dental treatment plan. It is furthernoted that an “estimated outcome of an orthodontic treatment plan” mayinclude an estimate of a state of a patient's dentition after correctionof any malpositioned teeth/jaws, malocclusion, etc., the patient suffersfrom. In some implementations, an estimated outcome of an orthodontictreatment plan includes the estimated state of the patient's dentitionif the patient's dentition has been changed to have a model and/or idealarch form, as reflected according to one or more databases of historicalcases and/or idealized arch forms. An “actual outcome of an orthodontictreatment plan” may represent the state of a patient's dentition afterimplementation of the orthodontic treatment plan; an “intended outcomeof an orthodontic treatment plan” may represent an intended state of apatient's dentition after implementation of an orthodontic treatmentplan.

A better understanding of the features and advantages of the presentdisclosure will be obtained by reference to the following detaileddescription that sets forth illustrative embodiments, in which theprinciples of embodiments of the present disclosure are utilized, andthe accompanying drawings.

FIG. 1 illustrates an example of a method 100 of providing an estimatedand/or intended outcome of a dental treatment plan, in accordance withone or more embodiments herein. The method 100 may be executed by any ofthe systems disclosed herein. It is noted that various examples mayinclude more or fewer blocks than those depicted in FIG. 1.

At block 110, one or more two-dimensional (2D or 2-D) image(s) of apatient are captured. In some embodiments, the 2D image(s) depict themouth of the patient and include one or more images of the face, head,neck, shoulders, torso, or the entire patient. The 2D image(s) of thepatient may include an image of the patient with the patient's mouth inone or more positions; for example, the patient's mouth may be a smilingposition, such as a social smiling position, a repose position withrelaxed muscles and lips slightly parted, or anterior retracted openbite or closed bite positions.

In some embodiments, the image of the patient is obtained with an imagecapture device. An “image capture device” (i.e., “image capture system”)as used herein, may include any system capable of capturing an image.Examples of image captures devices include a camera, smartphone, digitalimaging device, a component of a computer system configured to captureimages, etc. The image may be captured with a lens at a predeterminedfocal length and at a distance from the patient. The image may becaptured remotely and then received for processing. In someimplementations, the image of the patient is obtained from acomputer-storage device, a network location, a social media account,etc. The image may be a series of images or video captured from one ormore perspectives. For example, the images may include one or more of afrontal facial image and a profile image, including one or morethree-quarter profile images and full profile images.

At block 120, a three-dimensional (3D or 3-D) model of the patient'steeth is generated based on the 2D image of the patient. As discussed inmore detail with respect to FIG. 6 and elsewhere herein, generating a 3Dmodel may include identifying the patient's teeth. Generating a 3D modelmay further include identifying a patient's gums, lips, and/or mouthopening, forming a parametric model for each identified tooth, andcombining the parametric teeth models into one or more (e.g., two,three, etc.) parametric arch models. The parametric models of thepatient's teeth and arch may be based on or in reference to meanparametric tooth and arch models, as discussed further herein.

A “parametric model of a patient's teeth” (e.g., a “parametric model ofa patient's dentition”) as used herein, may include a model of apatient's dentition (e.g., a statistical model) characterized by aprobability distribution with a finite number of parameters. Aparametric model of a patient's dentition may include a parametric modelof the patient's teeth and/or arch. A parametric model may comprise amodel representing objects in various dimensions, and may include aparametric 2D model, a parametric 3D model, etc. Use of a parametricmodel of a patient's dentition to model the patient's teeth may reducethe memory and computational demands when manipulating, comparing, orotherwise using digital models, as descried herein and simplifycomparisons between different models. In some implementations, aparametric model of a tooth can be expressed as:

S _(Σ) ^(C)+Σ_(i) ^(|B) ^(τ) ^(|) a _(τ) ^(i) B _(τ) ^(i)  eq. (1),

where S_(Σ) ^(C) is a mean tooth shape, a generic parameter. Each tooth(for example, tooth number 6, the upper right canine in the universaltooth numbering system, has its own mean tooth shape) calculated fromthousands of teeth with same tooth number, as discussed herein. Thesymbol τ may represent the index for each tooth, which may be based onthe universal tooth numbering system or another numbering system, B_(τ)^(i) may be the principal components of the shape of each tooth, also ageneric parameter, and a_(τ) ^(i) may be the coefficients for theprincipal components of the shape of the teeth, a case-specificparameter. Accordingly, eq. (1) can be used to express a parametricmodel of each of a specific patient's teeth based on each tooth'sparticular shape relative to a mean tooth shape for that tooth (forexample, a left lower incisor, right upper canine, etc.).

A parametric model of a patient's dentition can be expressed as:

Z _(τ) =ΦTT _(τ) T _(τ) ^(C)(S _(τ) ^(C)+Σ_(i) ^(|B) ^(τ) ^(|) a _(τ)^(i) B _(τ) ^(i))  eq. (2),

where S_(Σ) ^(C)+Σ_(i) ^(|B) ^(τ) ^(|)a_(τ) ^(i)B_(τ) ^(i) is asdiscussed above with reference to eq. (1); T_(τ) ^(C) is the mean toothposition, a generic parameter; T is the deviation of the position of apatient's tooth from the corresponding mean tooth position, acase-specific parameter; and Φ is an arch scaling factor that scales theparametric, unit-less, values to real world values, also a case-specificparameter. T is the global perspective of the arch from a viewpoint andis a case-specific parameter and in some embodiments, is only used whenmatching to a 2D image, because arch scans typically do not have aperspective, whereas a 2D image, such as a camera does.

To generate a parametric 3D model of a patient's tooth from a 3D toothmodel derived from an image of a patient's tooth or from another methodknown in the art, the tooth may be modeled based on displacement of thescanned tooth surface from a fixed shape, such as a fixed sphere. Toillustrate this point, reference is made to FIG. 2, illustrating aparametric tooth model, in accordance with one or more embodimentsherein. In the example of FIG. 2, a sphere 210 having a plurality ofvertices in fixed or known locations and orientations is shown. A tooth220 may be placed or otherwise modeled at the center of the sphere 210.In some embodiments, the center of volume of the tooth 220, the scannedportion of the tooth 220, or the crown of the tooth 220, may be alignedwith the center of the sphere 210. Then each vertex 230 a, 230 b of thesphere 210 may be mapped to a location on the surface of the toothmodel. In some embodiments, the mapping may be represented by an n*3matrix, where n represents the number of points on the sphere, such as2500, and then for each point, the x, y, and z location is recorded. Insome embodiments, the matrix stores the difference between a location ona mean tooth with the corresponding position on the model of the actual.In this way, each tooth is represented by the same 2500 points, anddifferences between the teeth are easily compared with each other. Thedifference may be represented using PCA components as Σ_(i) ^(|B) ^(τ)^(|)a_(τ) ^(i)B_(τ) ^(i), wherein each specific case eventually has aunique set of a_(τ) ^(i), since B_(τ) ^(i) is generic for all cases. Toillustrate this point, reference is made to FIG. 3A, showing an exampleof how well parametric models 320 of the teeth match the original 3Dmodels 310. The parameters of a particular tooth may be stored in adatastore, such as a database and may be represented by a_(τ) ^(i).

A parametric model of a patient's arch may involve parameterizing thelocation and orientation of each tooth in an arch and the scale of thearch. The case-specific parameters for a particular tooth may be storedin a matrix or other datastore, as represented by eq. (3):

$\begin{matrix}{{T_{\tau} = \begin{bmatrix}\xi_{00} & \xi_{01} & \xi_{02} & \Delta_{\tau,x} \\\xi_{10} & \xi_{11} & \xi_{12} & \Delta_{\tau,y} \\\xi_{20} & \xi_{21} & \xi_{22} & \Delta_{\tau,z} \\0 & 0 & 0 & 1\end{bmatrix}};} & {{eq}.\mspace{11mu} (3)}\end{matrix}$

where τ_(ij) are the rotational components that define the orientationof the tooth with respect to the arch and may be a function of α_(τ),β_(τ), γ_(τ), the 3 rotation angles of the tooth. Δ_(τ,x), Δ_(τ,y), andΔ_(τ,z) are the translation of the center point of the tooth withrespect to the arch origin. The rotation may be based on the orientationof orthogonal axis of each tooth, for example, the long axis, thebuccal-lingual axis, and the mesial-distal axis with respect to a fixedreference or with respect to a mean rotation. In some embodiments one ormore of these components may be represented as deviations or changesrelative to a mean arch, discussed herein.

Similarly, the scaling factor, Φ is applied to the teeth and the archlocations of the teeth to scale the arch from a generic or unit-lessrepresentation to a real world scale that represents the actual size andlocation of the teeth in the arch. In some embodiments, scaling may bebetween projected 3D units, for example, millimeter, to images size, forexample, pixels.

As discussed above, and elsewhere herein, the parametric models of thearch, including the teeth, may be represented based on a mean archmodel, including the teeth. The mean arch model may be determined basedon an average of a large dataset of patient arch scans. In someembodiments, the mean arch model may be based on a large dataset ofparameterized arches.

For example, a mean arch may be built from a set of previously scannedand/or segmented arches. FIG. 3B depicts an example of a method 350 ofdetermining a mean arch model and is discussed in detail further herein.

In some implementations, a parametric model of an arch may be convertedinto a 3D model of the arch. FIG. 5 depicts a method 500 of converting aparametric model of one or more teeth into a 3D model of a dental arch,according to some implementations, and is discussed in detail furtherherein.

Returning to FIG. 1, at block 130, an estimated and/or intended outcomeof a dental treatment plan on the 3D model is identified to obtain anestimated and/or intended outcome of the orthodontic treatment plan. Theestimated and/or intended outcome of a dental treatment plan may bebased on a parametric model of the patient's teeth at positionsindicated based on a predetermined arch model, as discussed with respectto FIGS. 9-11. In some embodiments, a predetermined arch model may bebased on a mean arch model may be based on a historic mean of collectedarch scans or other arch models from patients. In some embodiments, thepredetermined arch model may be an idealized model, for example, basedon a clinically ideal arch or an arch model wherein tooth positionsand/or orientations are determined in advance, such as based on one ormore of their aesthetic and clinical properties, such as properocclusion. In some embodiments, an estimated and/or intended outcome ofthe dental treatment plan may correspond to an intended final position(or estimate thereof) of the patient's teeth after implementation of thetreatment plan. In some embodiments, the estimated and/or intendedoutcome of dental treatment may correspond to estimates of final andintermediate positions during the course of the treatment plan. As notedherein, the dental treatment plan may comprise an orthodontic treatmentplan, a restorative treatment plan, some combination thereof, etc.

At block 140, second 2D image(s) are generated showing the estimatedand/or intended outcome of orthodontic treatment. The image(s) may be a2D facial image of the patient with teeth aligned according theestimated and/or intended outcome of the dental treatment plan asdiscussed herein. In some embodiments, the image may include anestimated and/or projected texture of the patient's teeth, for exampleas discussed below, with respect to FIG. 9 and elsewhere herein. A“projected texture” or “estimated texture” of a patient's teeth, as usedherein, may include a projection/estimation of texture of teeth and mayinclude the feel, appearance, consistency, and/or other attributes ofthe surface of the teeth. At block 150, the second 2D image(s) generatedat block 140 are provided to a user. For example, the image may berendered for viewing by a patient or dental professional. In someembodiments, the second 2D image may be loaded into memory or retrievedfrom memory.

Turning to FIG. 3B, FIG. 3B illustrates a method of determining genericparameters from historic and/or idealized cases, in accordance with oneor more embodiments herein. At block 360, historic and/or idealizedcases are acquired. The historic and/or idealized cases may be retrievedfrom a datastore. The historic and/or idealized cases may include casesrepresenting previously scanned and segmented arch models. In someimplementations, the historic and/or idealized cases may represent archmodels of treated patients (e.g., of patients who in the past haveundergone treatment) and/or idealized arch models representing intendedoutcomes of various forms of orthodontic treatment. In variousimplementations, the historic and/or idealized cases may represent archmodels with idealized arch forms. In some implementations, the historicand/or idealized cases may include regions where model arches have teeththat correspond to a location of an implant to be inserted into the archof a patient.

At block 370 the historic and/or idealized cases are aligned. In someimplementations, each arch of the historical and/or idealized cases isaligned at a plurality of locations. As an example, each arch of thehistorical and/or idealized cases may be aligned at the following threelocations: between the central incisors and at each distal end of theleft and right side of each arch. For example, FIG. 4 shows a set ofarches 400 aligned at location between the central incisors 410, at theleft distal end of the arch 430, and the right distal end of the arch420. It is noted the historic and/or idealized cases may be aligned at avariety of locations and a variety of numbers of locations withoutdeparting from the scope and substance of the inventive conceptsdescribed herein.

Returning to FIG. 3B, determining a mean arch model and determining archmodel distributions may include performing sub operations at block 370.For example, each arch is averaged to determine T_(τ) ^(C), then theindividual tooth local tranformations are determined and compared toT_(τ) ^(C) to determine T_(τ), then after aligning each tooth β_(τ) maybe determined.

At block 380, the distribution estimation for case-specific parametersare determined. For example, each tooth's relative shape, location, androtation are determined in order to build a distribution for eachcase-specific parameter. For example the distribution of the a_(τ) ^(i),the coefficients for the principal components for the surface model ofthe each respective tooth in all of the retrieved models, is determined.

The location and orientation of each tooth may be averaged to determinea mean location for each tooth and the orientation of each tooth isaveraged to determine a mean orientation for each tooth. The meanlocation and orientation are used to determine T_(τ) ^(C).

At block 390, the mean of the distribution of the estimation forcase-specific parameters is used as the generic parameters of mean toothposition and mean tooth shape.

FIG. 5 depicts a method 500 of generating a parametric model of one ormore of a patient's teeth and converting the parametric model of one ormore teeth into a 3D model of a dental arch, according to someimplementations. The method 500 may be used for generating a 3D model ofa tooth arch of a patient based on a parametric model of the patient'steeth.

At block 510, a mean shape of each tooth is determined. As an example,the mean shape, S_(τ) ^(C), of each tooth is determined. In someembodiments, the mean shape may be based on the mean shape of a set ofarches from historical and/or idealized cases, as discussed herein. Insome embodiments, the mean shape may be rendered, for example, on ascreen for viewing by a patient or dental professional. For example,mean tooth shapes 512 are rendered for viewing. In some embodiments, themean shape may be loaded into memory or retrieved from memory. The meanshape may also be initialized as a set of matrices, one for each tooth.

At block 520, a principal component analysis shape adjustment isperformed on the mean shape of the teeth. This adjustment adjusts theshape of the teeth based on the patient's particular teeth, for example,based on a scan of the patient's teeth, a 2D image, or other imagingtechnologies, as discussed herein. As an example, a principal componentanalysis shape adjustment is performed on the mean shape of the teeth,S_(τ) ^(C). For each tooth in the model, the case-specific coefficientsfor the principal components, a_(τ) ^(i), are applied to the principalcomponent, B_(τ) ^(i). After the shape adjustment is completed, in someembodiments, the adjusted shape may be rendered, for example, on ascreen for viewing by a patient or dental professional. For example,adjusted tooth shapes 522 are rendered for viewing. In some embodiments,the adjusted shape may be stored into memory. The adjusted shape mayalso be stored as a set of matrices, one for each tooth.

At block 530, a mean tooth pose is determined. In some embodiments, themean tooth pose may be based on the mean tooth pose of a set of archesfrom historical and/or idealized case, as discussed herein. In someembodiments, at block 530, each adjusted tooth 522 is placed in itscorresponding mean location and orientation, as determined by the meanarch. In some embodiments, the mean tooth pose may be rendered, forexample, on a screen for viewing by a patient or dental professional.For example, mean tooth pose 532 is rendered for viewing. In someembodiments, the mean tooth pose may be loaded into memory or retrievedfrom memory. The mean tooth pose may also be initialized as a set ofmatrices, one for each tooth in the tooth pose. In some embodiments, themean tooth shapes from block 510 may be placed in their correspondingmean tooth pose at block 530 before the shapes of the teeth are adjustedat block 520. In other words, the order of block 520 and block 530 maybe swapped.

At block 540, a tooth pose adjustment is performed on the mean toothpose. This adjustment adjusts the shape of the teeth based on thepatient's particular teeth, for example, based on a scan of thepatient's teeth, a 2D image, or other imaging technologies, as discussedherein. In some embodiments, the pose adjustment T_(τ) is based on theparticular tooth pose of a patient's arches, as discussed above. In someembodiments, at block 540, the position and orientation of each tooth522 is adjusted such that it is placed in a location and orientation asdetermined by the location and orientation of the teeth in a patient'simaged arch, or otherwise, as discussed herein. In some embodiments, theadjusted tooth pose may be rendered, for example, on a screen forviewing by a patient or dental professional. For example, adjusted toothpose 542 is rendered for viewing. In some embodiments, the adjustedtooth pose may be stored in memory. The adjusted tooth pose may also bestored as a set of matrices and/or data structures, e.g., one for eachtooth in the tooth pose. In some embodiments, the mean tooth shapes fromblock 510 may be placed in their corresponding adjusted tooth pose atblock 540 before the shapes of the teeth are adjusted at block 520. Inother words, the order of block 520 and blocks 530 and 540 may beswapped such that block 520 takes place after blocks 530 and 540.

At block 550, the arch is scaled such that it is generated according tothe patient's tooth and arch size. In some embodiments, the arch scalingfactor Φ is based on the particular tooth and arch of a particularpatient, as discussed above. In various embodiments, the arch scalingfactor may also be based on one or more of the image size of the 2Dimage of the patient, for example, when scaling the 3D model forintegration into a 2D image. In some embodiments, at block 550, the sizeof each tooth 522 and the arch is adjusted such that the scaled archmatches the size of a patient's arch, as determined, for example, bysize of the teeth and arch in the patient's imaged arch, or otherwise,as discussed herein. In some embodiments, the scaled arch may berendered, for example, on a screen for viewing by a patient or dentalprofessional. For example, scaled arch 552 is rendered for viewing. Insome embodiments, the scaled arch may be stored in memory. The scaledarch may also be stored as a set of matrices and/or data structures, onefor each tooth in the tooth pose. In some embodiments, the mean toothshapes from block 510 may be placed in their corresponding scaledposition and size at block 550 before the shapes of the teeth areadjusted at block 520. In other words, the order of block 520 and blocks530, 540, and 550 may be swapped such that block 520 takes place afterblocks 530, 540, and 550.

The blocks of method 500 may be carried out in orders other than thoseshown in FIG. 5. For example, blocks 510 and 530 may be performed beforeblocks 520, 540, and 550. In some embodiments, blocks 510, 520, 530, and550 may be carried out before block 540. These and other modificationsto the order of the blocks in method 500 may be carried out withoutdeviating from the spirit of the disclosure.

With attention to FIG. 6, FIG. 6 illustrates a method 600 forconstructing a 3D model from a 2D image, according to one or moreembodiments disclosed herein.

At block 610, a 2D image of a patient is captured. In some embodiments,the 2D image includes the mouth of the patient and one or more images ofthe face, head, neck, shoulders, torso, or the entire patient. The 2Dimage of the patient may include an image of the patient with theirmouth in one or more positions. For example, the patient's mouth may bea smiling position, such as a social smiling position, a repose positionwith relaxed muscles and lips slightly parted, or anterior retractedopen bite or closed bite positions.

In some embodiments, the image of the patient is taken with an imagecapture system. The image may be captured with a lens at a predeterminedfocal length and at a distance from the patient. The image may becaptured remotely and then received for processing. The image may begathered from a storage system, a networked location, a social mediawebsite, etc. The image may be a series of images of video captured fromone or more perspectives. For example, the images may include one ormore of a frontal facial image and a profile image, including one ormore three-quarter profile images and full profile images.

At block 620, edges of oral features of the patient are determined. Forexample, the edges of one or more of the patient's teeth, lips, andgingiva may be determined. An initial determination of the patient'slips (for example, the inner edges of the lips that define the mountopening), teeth, and gingiva contours may be identified by a machinelearning algorithm, such as a convoluted neural network. The machinelearning algorithm may be trained based on pre-identified labels of thelips, teeth, and gingiva visible within a 2D image of a patient. Theinitial contours may be weighted contours such that the machine learningalgorithm's confidence that a given position in the image, such as ateach pixel, is an edge or contour of a patient's lips, teeth, orgingiva.

The initial contours may be extracted from the image. The initialcontours may have a brightness or other scale applied to them. Forexample, in a grey scale image of the contours, each pixel may beassigned a value between 0 and 255, which may indicate a confidence thatthe pixel is a contour or may indicate the magnitude of the contour atthat location.

The pixels that denote the contours may then undergo binarization tochange the pixels from a scale of, for example, 0 to 255, to a binaryscale of, for example, 0 or 1, creating binarized tooth contours. In thebinarization process, the value of each pixel is compared to a thresholdvalue. If the value of the pixel is greater than the threshold, then itmay be assigned a new first value, such as a value of 1 and if the pixelis less than the threshold, then it may be assigned a new second value,such as a value of 0, for example.

The binarized tooth contours may be thinned, whereby the contour'sthickness is reduced to, for example, a single pixel in width, forming athinned contour. The width of a contour for thinning may be measured asthe shortest distance from a contour pixel adjacent to a non-contourpixel on a first side of a contour, to a contour pixel adjacent anon-contour pixel on a second side of the contour. The single pixelrepresenting the thinned contour at a particular location may be locatedat the midpoint of the width between the pixel at the first side and thepixel at the second side. After thinning the binarized tooth contours,the thinned contour may be a single width contour at a location thatcorresponds to a midpoint of the binarized contour.

At block 630, parameterized 3D tooth and arch models are matched to eachof the patient's teeth that are depicted in the 2D image of the patient.The matching may be based on the edges, also referred to as contours,determined at block 630. FIG. 7A shows an example of a process ofmatching tooth and arch models to the edges. Returning to FIG. 6, afteridentifying and modeling the teeth and arch, or as part of such aprocess, missing or broken teeth may be inserted into the parameterized3D tooth and arch model. For example, when a tooth is missing orsignificantly damaged, the parameterized model of that tooth may bereplaced with a mean tooth model, thereby simulating a prosthetic tooth,such as a veneer, crown, or implant prosthetic. In some embodiments, themissing tooth may be left as a space in the arch. In some embodiments abroken tooth may be left, as is, without replacing it with a mean shapetooth.

During the matching process, the case-specific parameters of aparametric arch model are varied and iterated through until a matchbetween the parametric arch model and the teeth depicted in the 2D imageis found. This match may be determined based on a projection of theedges of a silhouette of the parametric model match with the edges ofthe lips, teeth, and gingiva identified in the 2D image.

At block 640, the parametric model of the patient's teeth is rendered.An example of a process of rendering a parametric model of teeth isdescribed with respect to FIG. 8, and elsewhere herein. During therendering process, a 3D model of the patient's teeth is formed based ondata describing the teeth and arch of the patient. For example, based onthe parametric 3D model formed at block 630, or otherwise describeherein. In some embodiments, the 3D model is directly inserted into the2D image of the patient. In such embodiments, the actions in block 640may be omitted or merged with the actions of block 650 such that, forexample, the 3D tooth model is rendered in 2D form for insertion intothe 2D image.

Optionally, at block 640, simulated treatments or viewingcustomizations, such as gingiva line adjustment, jaw positionadjustment, missing tooth insertion, or broken tooth repair, can beapplied to the 3D model before rendering into 2D form. In someembodiments, the edges of the parametric model (for example, lips,gingiva line, missing or broken teeth) may be altered to displaysimulated results of cosmetic or other treatments and procedures or toadjust the 2D patient image for customized viewing. For example, a user,such as a dental professional or patient, may adjust the gingiva line tosimulate gum treatments. As another example, the user may opt to show,hide, or repair missing or chipped teeth to simulate tooth repair orreplacement procedures. In another example, the user may opt to adjustjaw position in the pre- or post-treatment simulated image to simulateappearances with the jaw open, closed, or partially open. A jaw positionparameter can be defined by a distance between a tooth surface in theupper jaw and a tooth surface in the lower jaw, such as the distancebetween the incisal surface of the upper central incisor in relation tothe incisal surface of the lower central incisor. The jaw positionparameter can be defined and varied by the user. For example, thedistance between the incisal surface of the upper central incisor andthe incisal surface of the lower central incisor may be varied between−5 mm (indicating, the lower incisors overlapping with the upperincisors by 5 mm) and 10 mm (indicating, a gap of 10 mm between theupper incisors and the lower incisors). The gingiva line can be adjustedby replacing the patient gingiva line mask shape with the mean gingivaline mask shape from the historic shape datastore. Display of gingivaline adjustment can be optionally selected or adjusted by the user.Missing or broken teeth can be replaced with a mean tooth shape from thehistoric shape datastore. Display of missing tooth replacement or brokentooth repair can be optionally selected by the user. Simulatedtreatments or viewing customizations may be rendered, for example, on ascreen for viewing by a user, patient, or dental professional.

At block 650, the 3D tooth model is inserted into the 2D image of thepatient. The inner lip edges determined at block 620 may be used todefine the outline of the patient's mouth in the 2D image. At block 650,the area of the 2D image defined by the mouth opening may be removed andthe 3D model may be placed behind the 2D image and within the mouthopening. In some embodiments, inserting the 3D tooth model into the 2Dimage further comprises rendering an image of a parameterized gingivaline in the 2D image. The parameterized gingiva line may be determined,for example, as shown and described, with respect to method 2001 of FIG.20.

At block 660, textures are applied to the teeth. An example of applyingtextures is discussed in more detail with respect to FIG. 9. In someimplementations, when applying textures to the teeth, the 2D image ofthe patient is projected onto a 3D model of the teeth, such as theparametric 3D model of the patient's teeth determined, for example, atblock 630. When projecting the 2D image into the 3D model, the pixels ateach location in the 2D image are assigned to a location on the 3Dmodel. In some embodiments, the pixel value or texture information from2D image is processed before being applied to 3D model. Othertechniques, such as inpainting, blurring, image processing filters,pix2pix transformation technologies, etc., can also be used to generatetextures for application to the surfaces of the 3D model. The projectedpixels at each location on a surface of the 3D model form a texture forthat model. For example, the pixels projected onto a particular tooth ofthe patient form a texture for that tooth model. This texture can beapplied to the 3D tooth model.

FIG. 7A depicts a method 700 of building a patient-specific parametricmodel of a patient's teeth according to some embodiments.

At block 710, a course alignment of each corresponding mean parametrictooth is aligned with a respective center of a tooth in the 2D image. Acenter of the parametric tooth may be determined based on the areacenter of a projection of the parametric tooth's silhouette. A center ofa tooth identified in the 2D image of the patient may be determinedbased on an area center of an area defined by tooth edges and acorresponding lip and/or gingiva edge. The respective centers of the 2Dimage tooth and the parametric tooth may be aligned before theexpectation step and maximization steps of blocks 720 and 730,respectively.

The method 700 may dynamically generate the parametric tooth models withlip and gingiva edges for matching with the teeth in the 2D image andblock 710. In addition or alternatively, the lip and gingiva edges maybeapplied and/or dynamically adjusted at any of blocks 720, 730, and 740.The 3D tooth models may be computed on the fly for varying lip andgingiva line placement. Using such models provides for a much moreaccurate parametric model than using gingiva lines alone.

When lip and gingiva information is applied at block 710, in someembodiments, only the portion of the parametric model beyond the lipand/or gingiva edges is used to determine the area center of the tooth.Turning to FIG. 7B, the figure depicts examples of tooth models withgingiva edges 760 and tooth models with lip edges 770. At any of blocks710, 720, 730, 740, the location of the gingiva and/or lip edges may bemodified to adjust the fit of the silhouette of the tooth with thevisible portion of a corresponding tooth in the 2D image.

The addition of the lip or gingiva edges of the teeth significantlyimprove the efficiency and operation of process. The result is that theprocess much more realistically matches teeth models to the 2D image andthe process works with a much wider variety of photos, as compared to aprocess that does not apply lip and gingiva edges to teeth models.

Returning to FIG. 7A, at block 720, an expectation step is performed. Insome implementations, an Expectation Management (EM) engine and/or anengine configured to create a 3D model performs block 720. At theexpectation step, a silhouette of the tooth is projected onto the 2Dimage and the edges of the silhouette are evaluated against the edges ofthe tooth in the 2D image as determined based on the lip, gingiva, andtooth edges. The evaluation may be a determination of the normal at alocation at the edge of the silhouette and the closest location in the2D image with a similar normal. Then the probability that the two edgesare the same is determined. This process may be repeated for eachlocation at the edge of the silhouette.

At block 730, a maximization step of the EM engine is performed. At themaximization step a small angle approximation is used in order toprovide for an analytical solution to the maximization. The small angleapproximation and analytical solution provides a much improved solutionas compared to other methods, such as the Gaussian-Newton iterativemethod. The small angle approximation significantly reduces computationtime and resolves to an accurate solution much faster.

Blocks 720 and 730 may be performed iteratively on a single parameter orsubset of parameters, performing the expectation step then themaximization step, then back to the expectation step and so on until athreshold of convergence for the single parameter or subset ofparameters is reached. Then the process may proceed to block 740.

At block 740, an optimization parameter or subset of parameters areadded to the parametric model. For example, optimization of theparametric model may begin with Φ and T, then after iterating throughthe EM blocks 720 and 730, additional parameters are added. For example,7″, may be added and then optimized through the EM blocks 720 and 730for additional iterations. The number of iterations before addingadditional parameters may vary. In some embodiments, the EM blocks 720and 730 may be iterated through 3, 5, 7, 10, 15, 20, or an arbitrarynumber (e.g., any integer) of times. Finally, a_(τ) ^(i) may be added tothe parametric model, which is processed though the EM blocks 720 and730 until convergence is reached. During this process, outliers may bedetermined and filtered out. In some embodiments, after block 740,rather than looping back to block 720 and proceeding directly to theexpectation step, the process 700 may loop back to block 710, where acoarse alignment procession is performed based on the updated parametricmodel.

At block 750 the parameters of a parametric model are output to anotherengine or even to a datastore for later retrieval. For example, arendering engine may retrieve the parameters for the parametric modelfor rendering, as described with respect to FIG. 8, and elsewhereherein.

FIG. 8 depicts a method 800 of rendering patient's teeth in an initialposition, using a parametric model of the patient's arch, in accordancewith one or more embodiments herein.

At block 810, the mean shape, S_(τ) ^(C), of each tooth is determined.In some embodiments, the mean shape may be based on the mean shape of aset of arches taken, e.g., from historical cases and/or thoserepresenting idealized arch forms, as discussed herein. In someembodiments, the mean shape may be rendered, for example, on a screenfor viewing by a patient or dental professional. In some embodiments,the mean shape may be loaded into memory or retrieved from memory. Themean shape may also be initialized as a set of matrices, one for eachtooth.

At block 820, a principal component analysis shape adjustment isperformed on the mean shape of the teeth, S_(τ) ^(C). For each tooth inthe model, the case specific coefficients for the principal components,a_(τ) ^(i), are applied to the principal component, B_(τ) ^(i). Afterthe shape adjustment is completed, in some embodiments, the adjustedshape may be rendered, for example, on a screen for viewing by a patientor dental professional. For example, adjusted tooth shapes are renderedfor viewing. In some embodiments, the adjusted shape may be stored intomemory. The adjusted shape may also be stored as a set of matrices, onefor each tooth.

At block 830, the mean tooth pose is determined. In some embodiments,the mean tooth pose may be based on the mean tooth pose of a set ofscanned arches, as discussed herein. In some embodiments, at block 830,each adjusted tooth is placed in its corresponding mean location andorientation, as determined by the mean arch. In some embodiments, themean tooth pose may be rendered, for example, on a screen for viewing bya patient or dental professional. In some embodiments, the mean toothpose may be loaded into memory or retrieved from memory. The mean toothpose may also be initialized as a set of matrices and/or other datastructures, e.g., one for each tooth in the tooth pose. In someembodiments, the mean tooth shapes from block 810 may be placed in theircorresponding mean tooth pose at block 830 before the shapes of theteeth are adjusted at block 820. In other words, the order of block 820and block 830 may be swapped.

At block 840, a tooth pose adjustment is performed on the mean toothpose. In some embodiments, the pose adjustment T_(τ) is based on theparticular tooth pose of a patient's arches, as discussed above. In someembodiments, at block 840, the position and orientation of each tooth isadjusted such that it is placed in a location and orientation asdetermined by the location and orientation of the teeth in patient'sarch, or otherwise, as discussed herein. In some embodiments, theadjusted tooth pose may be rendered, for example, on a screen forviewing by a patient or dental professional. In some embodiments, theadjusted tooth pose may be stored in memory. The adjusted tooth pose mayalso be stored as a set of matrices and/or other data structures, e.g.,one for each tooth in the tooth pose. In some embodiments, the meantooth shapes from block 810 may be placed in their correspondingadjusted tooth pose at block 840 before the shapes of the teeth areadjusted at block 820. In other words, the order of block 820 and blocks830 and 840 may be swapped such that block 820 takes place after blocks830 and 840.

At block 850, the arch is scaled such that it is generated with toothdimensions according to the patient's tooth and arch size. In someembodiments, an arch scaling factor Φ is based on the particular toothand arch of a particular patient, as discussed above. In someembodiments, at block 850, the size of the arch is adjusted such thatthe scaled arch matches the size of a patient's arch, as determined, forexample, by size of the teeth and arch in the patient's arch, orotherwise, as discussed herein. In some embodiments, the scaled arch maybe rendered, for example, on a screen for viewing by a patient or dentalprofessional. In some embodiments, the scaled arch may be stored inmemory. The scaled arch may also be stored as a set of matrices and/ordata structures, e.g., one for each tooth in the tooth pose. In someembodiments, the mean tooth shapes from block 810 may be placed in theircorresponding scaled position and size at block 850 before the shapes ofthe teeth are adjusted at block 820. In other words, the order of block820 and blocks 830, 840, and 850 may be swapped such that block 820takes place after blocks 830, 840, and 850.

The blocks of the method 800 may be carried out in orders other thanthose shown in FIG. 8. For example, blocks 810 and 830 may be performedbefore blocks 820, 840, and 850. In some embodiments, blocks 810, 820,830, and 850 may be carried out before block 840. These and othermodifications to the order of the blocks in method 800 may be carriedout without deviating from the spirit of the disclosure.

FIG. 9 depicts a method 900 of constructing and applying textures to a3D model, in accordance with one or more embodiments herein. Texturesmay aid in providing detail, such as color information, to the 3D modelof a patient's teeth. As described herein, the process 900 uses imagesof the patient's teeth to provide lifelike textures for a patient'steeth.

Accordingly, at block 910, a 3D model of the patient's teeth and a 2Dimage of the patient are acquired, as described elsewhere herein, forexample, in the discussion related to FIG. 5. The 2D image and the 3Dmodel should depict the teeth in the same positions, for example, the 3Dmodel could be a parametric model derived from the 2D image of thepatient.

At block 920, the 2D image of the patient's teeth is projected onto the3D model and the image is aligned with the model. Such alignment may becarried out by matching contours in the 3D model with contours in the 2Dimage. Contours may include lip, gingiva, and teeth edges, determined asdescribed elsewhere herein.

At block 930, the color information from the 2D image is mapped astextures to the 3D model. Color information may include lightingconditions, such as specular highlights, accurate tooth coloration andtextures, and other tooth features, such as tooth jewelry, gold teeth,etc. Once the textures are mapped to the 3D model, the teeth in the 3Dmodel may be repositioned, for example to depict an estimated and/orintended outcome of a dental treatment plan (e.g., an orthodontictreatment plan, a restorative treatment plan, some combination, thereof,etc.). Such a final position includes both accurate color and toothfeature information and accurate 3D positioning of the patient's teethand may be used for example, in simulating an estimated and/or intendedoutcome of a dental treatment plan (e.g., a final orthodontic position)of a patient's teeth, as described with respect to FIG. 10A, FIG. 10B,and/or elsewhere herein.

In some embodiments, the textures model is adjusted to simulate clinicalor beautification treatments. For example, color may be adjusted tosimulate tooth whitening procedures. Simulating such treatments mayinclude generating a mask from a 2D projection of the 3D tooth modelwith the model in an arrangement for insertion in the 2D image, forexample, in either the initial or final positions. The mask can beapplied to the 2D image of the patient's teeth in either the initial orfinal positions. Color adjustment or whitening can be further applied tothe mask region. Color adjustment and whitening parameters can beoptionally selected and adjusted by the user. Color adjustments andwhitening may be rendered, for example, on a screen for viewing by auser, patient, or dental professional.

FIG. 10A depicts a method 1000 for simulating an estimated and/orintended outcome of a dental treatment plan on a patient's teeth, inaccordance with one or more embodiments herein.

At block 1010, a model of a mean set of teeth arches is built. In someembodiments, the model is a parametric 3D model of a mean arch based ona set of historic scanned arches and/or arches representing idealizedarch forms. In some embodiments, the scans are from the initialpositions of patient's teeth. In some embodiments the scans are frompatients after they have completed orthodontic treatment. In still otherembodiments, the scans are taken without regard to whether the patienthas undergone orthodontic treatment or not. In some implementations, thehistoric and/or idealized cases may represent arch models of treatedpatients (e.g., of patients who in the past have undergone treatment)and/or idealized arch models representing intended outcomes of variousforms of orthodontic treatment. Block 1010 may include the process 350as described with respect to FIG. 3B. At block 1010, cases are acquired.The cases may be retrieved from a datastore and may include previouslyscanned and segmented arch models.

The arches may also be aligned. Each arch in the datastore may alignedat a plurality of locations. For instance, each arch may be aligned atthe following three locations: between the central incisors and at eachdistal end of the left and right side of each arch. For example, FIG. 4shows a set of arches 400 aligned at location between the centralincisors 410, at the left distal end of the arch 430, and the rightdistal end of the arch 420.

In some embodiments, determining a mean arch model and determining archmodel distributions includes performing sub steps. For example, eachhistoric arch may be rescaled to determine Φ, then each arch is averagedto determine T_(τ) ^(C), then the individual tooth local transformationsare determined and compared to T_(τ) ^(C) determine T_(τ), then afteraligning each tooth β_(τ) is determined.

The distribution estimation for case-specific parameters are alsodetermined. For example, each tooth's relative shape, location, androtation are determined in order to build the distribution for eachcase-specific parameter. For example the distribution of the a_(τ) ^(i),the coefficients for the principal components, for the surface model ofthe each respective tooth in all of the retrieved models is determined.

The location and orientation of each tooth is averaged to determine amean location for each tooth and the orientation of each tooth isaveraged to determine a mean orientation for each tooth. The meanlocation and orientation are used to determine T_(τ) ^(C).

Finally, the mean of the distribution of the estimation forcase-specific parameters may be used as the generic parameters of meantooth position and mean tooth shape.

At block 1020, a 2D image of a patient is captured. In some embodiments,the 2D image includes the mouth of the patient and one or more images ofthe face, head, neck, shoulders, torso, or the entire patient. The 2Dimage of the patient may include an image of the patient with theirmouth in one or more positions; for example, the patient's mouth may bea smiling position, such as a social smiling position, a repose positionwith relaxed muscles and lips slightly parted, or anterior retractedopen bite or closed bite positions.

In some embodiments, the image of the patient is obtained, e.g., takenwith an image capture system. The image may be captured with a lens at apredetermined focal length and at a distance from the patient. In someimplementations, the image of the patient is obtained from acomputer-storage device, a network location, a social media account,etc. The image may be captured remotely and then received forprocessing. The image may be a series of images of video captured fromone or more perspectives. For example, the images may include one ormore of a frontal facial image and a profile image, including one ormore three-quarter profile images and full profile images.

At block 1030, a 3D model of the patient's teeth is constructed from the2D image of the patient's teeth. The 3D model may be based on aparametric model constructed according to the process described withrespect to FIG. 6, wherein the edges of the oral features of the patientare determined. For example, the edges of one or more of the patient'steeth, lips, and gingiva may be determined. An initial determination ofthe patient's lips (for example, the inner edges of the lips that definethe mount opening), teeth, and gingiva contours may be identified by amachine learning algorithm, such as a convoluted neural network.

Then, the initial contours may be extracted from the image. The pixelsthat denote the contours may then undergo binarization. The binarizedtooth contours are thinned, whereby the contour's thickness is reducedto, for example, a single pixel in width, forming a thinned contour.

Using the contours, either thinned or otherwise, parameterized 3D teethand arch models are matched to each of the patient's teeth that aredepicted in the 2D image of the patient. The matching is based on thecontours, determined above. After identifying and modeling the teeth andarch, or as part of such a process, missing or broken teeth may beinserted into the parameterized 3D tooth and arch model.

At block 1040, case-specific parameters are estimated. In someembodiments, the case-specific parameters may be determined at block1030 as part of the model construction according to the processdescribed, for example, at FIG. 6. During the matching process or atblock 1040, the case-specific parameters of a parametric arch model arevaried and iterated through until a match between the parametric archmodel and the teeth depicted in the 2D image is found. This match may bedetermined based on a projection of the edges of a silhouette of theparametric model match with the edges of the lips, teeth, and gingivaidentified in the 2D image, as described herein.

At block 1050, the patient's teeth are rendered according to a simulatedoutcome of a dental treatment plan using the patient's case-specificparameters for tooth shape or arch scale, but using the mean value fortooth position. FIG. 11 shows an example of a method for rendering teethaccording to an and/or intended outcome of a dental treatment plan usingthe appropriate case-specific and mean parameter values.

Optionally, at block 1050, other changes, such as gingiva lineadjustment, jaw position adjustment, missing tooth replacement, orbroken tooth repair, can be applied to the 3D model before renderinginto 2D form. In some embodiments, the edges and other features of themodel, such as lips, gingiva line, and missing or broken teeth, may bealtered to display simulated results of treatment procedures or toadjust the 2D patient image for customized viewing. For example, theuser may adjust the gingiva line to simulate gum treatments. In anotherexample, the user may opt to show, hide, or repair missing or chippedteeth to simulate tooth repair or replacement procedures. In this casean ideal tooth, based on the ideal parameters discussed above, may beused to replace the missing or damaged tooth. The missing or damagedtooth may be replaced based on one of the teeth in the historicdatastore, or with a patient's own tooth, such as a corresponding toothfrom the opposite side of the patient's arch may be used. For example,if the left upper canine is missing or chipped, a mirrored model of thepatient's right upper canine may be used in the position of the missingleft upper canine. In another example, the user may opt to adjust jawposition in the pre- or post-treatment simulated image to simulateappearances with the jaw open, closed, or partially open. A jaw positionparameter can be defined by distance between a tooth surface in theupper jaw and a tooth surface in the lower jaw, such as the distancebetween the incisal surface of the upper central incisor in relation tothe incisal surface of the lower central incisor. The jaw positionparameter can be defined and varied by the user. The gingiva line can beadjusted by replacing the patient gingiva line mask shape with the meangingiva line mask shape from the historic mean shape database. Displayof gingiva line adjustment can be optionally selected or adjusted by theuser. Missing or broken teeth can be replaced with a mean tooth shapefrom the historic mean shape database. Display of missing toothreplacement or broken tooth repair can be optionally selected by theuser. Simulated treatments or viewing customizations may be rendered,for example, on a screen for viewing by the user, a patient, or a dentalprofessional.

At block 1060, the 3D model is inserted into 2D image of the patient.For example, the 3D rendering of the teeth may be placed in the mouth ofthe patient as defined by the lip contours determined at block 1030 aspart of the model construction process. In some embodiments, insertingthe 3D tooth model into the 2D image further comprises rendering animage of a parameterized gingiva line in the 2D image. The parameterizedgingiva line may be determined, for example, as shown and described,with respect to method 2001 of FIG. 20. In some embodiments, at block1060 the textures model is adjusted to simulate clinical beautificationtreatments. For example, color may be adjusted to simulate toothwhitening procedures. A mask can be generated from at 2D projection ofthe 3D tooth model. The mask can be applied to the 2D image of thepatient's teeth, in either the initial or final positions. Coloradjustment or whitening can be further applied to the mask region. Coloradjustment and whitening parameters can be optionally selected andadjusted by the user.

Simulated treatments or viewing customizations performed at block 1050or block 1060 can be selected, de-selected, or adjusted by the user onthe projected 2D image. Parameters can be adjusted to simulatetreatments, such as whitening, gum line treatments, tooth replacement,or tooth repair, or for viewing customization, such as to display jawopen, closed, or partially open. In one example, the user can optionallyadjust a color parameter to simulate tooth whitening. The user canadjust the color parameter to increase or decrease the degree of toothwhitening. In another example, the user can optionally adjust a gingivaline parameter to simulate gum line treatments or changes as a result ofdental hygiene. The user can adjust the gingiva line parameter to raisethe gingiva line (decrease the amount of exposed tooth surface) tosimulate gum recovery, such as resulting from improved dental hygiene orgum line treatments, or lower the gingiva line (increase the amount ofexposed tooth surface) to simulate gum recession, such as resulting frompoor dental hygiene. In another example, the user can optionally selecta tooth replacement parameter. The user can select the tooth replacementparameter to replace one or more missing, broken, or damaged teeth withan ideal tooth, such as with a tooth from the historic datastore or witha patient's own tooth. In another example, the user can optionallyadjust a jaw line parameter for customized viewing of jaw position. Theuser can increase a distance between a tooth in the upper jaw inrelation to a tooth in the lower jaw to simulate jaw opening or decreasethe distance between the tooth in the upper jaw in relation to the toothin the lower jaw to simulate jaw closing. In this case, the user cansimulate the appearance of orthodontic, cosmetic, or clinical treatmentsin a variety of jaw positions.

FIG. 10B depicts a process 1005 for simulating a final treatmentposition of a patient's teeth using a matched arch identified from aparameterized search of the treatment plan datastore.

At block 1015, a 2D image of a patient is captured, as described atblock 1020 of FIG. 10A, and, at block 1025, the 3D model of thepatient's teeth is constructed from the 2D image of the patient's teethas at block 1030 of FIG. 10A. At block 1035 case-specific parameters areestimated as at block 1040 of FIG. 10A.

At block 1045 the tooth shape parameters of the patient's teeth from the3D model are used to perform a parameterized search of the treatmentplan datastore. When matching the shapes of teeth of the patient withthe shapes of teeth of historic treatment in the datastore, the shapesof the teeth in the patient's parameterized 3D model are compared to theparameterized shapes of the teeth in each of the historic records in thetreatment datastore. When a match between the teeth shapes is found, thefinal tooth positions and orientations, or the final arch model of thematch record may be used in the simulated treatment plan. Aparameterized template arch may be identified based on comparison oftooth shape parameters to identify a matched template arch. In someembodiments, the match may be based on, for example, the closest matchtemplate arch. In some embodiments, the template shape may be loadedinto memory or retrieved from memory. The template shape may also beinitialized as a set of matrices, one for each tooth. Once a match isfound in a record within treatment datastore, the final arch model fromthe match record is retrieved and may be used as a basis for thepatient's target final tooth positions.

In some embodiments, the tooth locations and orientations in the matchedrecord are used as the basis for the tooth locations and orientationsfor the patient's target final tooth positions. When using the toothlocations and orientations, the patient's parameterized arch model maybe modified with the tooth locations and orientations in the matchedrecord or the match record may be updated with the shape of thepatient's teeth. In some embodiments, the model of each of the patient'steeth with an unaltered tooth shape is placed in the final tooth posedetermined from the matched record. In some embodiments, the model ofeach of the patient's teeth with an adjusted shape, such as the teethshapes from the matched record, is placed in the final tooth posedetermined from the matched record. Optionally, a tooth pose adjustmentcan be performed to adjust the positions of the teeth in the final toothpose.

In some embodiments, the final arch model from the matched record isused as the simulated final position for the patient's treatment in the2D rendering. In some embodiments, the teeth of the final arch modelfrom the matched record are placed in positions according to thepositions of the teeth in the patient's 2D photo. Alternately, the teethof the final arch model are placed in positions according to thepositions of the teeth in the patient's parameterized 3D model. Thefinal arch model, optionally with positions adjusted according to thepatient's 2D photo or 3D model, may be used as the simulated position inthe 2D rendering.

The shape of the template teeth may be adjusted based on the patient'sparameterized 3D model to more closely match the patient's tooth shapes.Optionally, a principal component analysis shape adjustment can beperformed on the template teeth shape. After the shape adjustment iscompleted, in some embodiments, the adjusted shape may be rendered, forexample, on a screen for viewing by a user, patient, or dentalprofessional. In some embodiments, the adjusted shape may be stored intomemory. The adjusted shape may also be stored as a set of matrices, onefor each tooth. In some embodiments, the patient's teeth are substitutedfor the teeth in the matching template from the treatment datastore.

The template arch may be scaled such that it is generated with accordingto the patient's tooth and arch size. In some embodiments, the templatearch scaling factor Φ is based on the particular tooth and arch of aparticular patient, as discussed above. In some embodiments, thetemplate arch scaling factor may also be based on one or more of theimage size of the 2D image of the patient, for example, when scaling the3D model for integration into a 2D image. In some embodiments, the sizeof each template tooth and the template arch is adjusted such that thescaled template arch matches the size of a patient's arch as determined,for example, by the size of the teeth and arch in the patient's scannedarch, or otherwise, as discussed herein. In some embodiments, the scaledtemplate arch may be rendered, for example, on a screen for viewing by auser, patient, or dental professional. In some embodiments, the scaledtemplate arch may be stored in memory. The scaled template arch may alsobe stored as a set of matrices, one for each tooth in the tooth pose. Insome embodiments, the ideal template tooth shapes may be placed in theircorresponding scaled position and size before the shapes of the teethare adjusted.

At block 1045, simulated treatments or viewing customizations, such asgingiva line adjustment, jaw position, missing tooth insertion, orbroken tooth repair, can be applied to the 3D model before renderinginto 2D form. Simulated treatments or viewing customizations changes maybe applied and adjusted as described at block 640.

At block 1055 the 3D model is inserted into a 2D image of the patient.For example, the 3D rendering of the teeth may be placed in the mouth ofthe patient as defined by the lip contours determined at block 1025 aspart of the model construction process. Rendering may be performed withor without the simulated treatments or viewing customizationsimplemented at block 1045. In some embodiments, inserting the 3D toothmodel into the 2D image further comprises rendering an image of aparameterized gingiva line in the 2D image. The parameterized gingivaline may be determined, for example, as shown and described with respectto method 2001 of FIG. 20. In some embodiments, at block 1055 thetextures model is adjusted to simulate clinical treatments. For example,color may be adjusted to simulate tooth whitening procedures. A mask canbe generated from a 2D projection of the 3D tooth model. The mask can beapplied to the 2D image of the patient's teeth, in either the initial orfinal positions. Color adjustment or whitening can be further applied tothe mask region. Color adjustment and whitening parameters can beoptionally selected and adjusted by the user.

Simulated treatments and viewing customizations performed at block 1045or block 1055 can be selected, de-selected, or adjusted by the user onthe projected 2D image. Parameters can be adjusted to simulatetreatment, such as whitening, gum line treatments, tooth replacement, ortooth repair, or for viewing customization, such as display jaw open,closed, or partially open.

FIG. 11 shows an example of a method 1100 for rendering teeth accordingto an estimated and/or intended outcome of a dental treatment plan basedon a parametric model of the patient's arch using patient-derived valuesfor the case-specific parameters of scale and tooth shape, but using amean value (e.g., one derived from historical and/or idealized arches)for tooth position.

At block 1110 the mean shape, S_(τ) ^(C), of each tooth is determined.In some embodiments, the mean shape may be based on the mean shape of aset of scanned arches, as discussed herein. In some embodiments, themean shape may be rendered, for example, on a screen for viewing by apatient or dental professional. In some embodiments, the mean shape maybe loaded into memory or retrieved from memory. The mean shape may alsobe initialized as a set of matrices, one for each tooth.

At block 1120, a principal component analysis shape adjustment isperformed on the mean shape of the teeth, S_(τ) ^(C). For each tooth inthe model, the case specific coefficients for the principal components,a_(τ) ^(i), are applied to the principal component, B_(τ) ^(i). Afterthe shape adjustment is completed, in some embodiments, the adjustedshape may be rendered, for example, on a screen for viewing by a patientor dental professional. In some embodiments, the adjusted shape may bestored into memory. The adjusted shape may also be stored as a set ofmatrices and/or data structures, e.g., one for each tooth.

At block 1130, the mean tooth pose is determined. In some embodiments,the mean tooth pose may be based on the mean tooth pose of a set ofscanned arches, as discussed above. In some embodiments, at block 1130,each adjusted tooth is placed in its corresponding mean location andorientation, as determined by the mean arch. In some embodiments, themean tooth pose may be rendered, for example, on a screen for viewing bya patient or dental professional. In some embodiments, the mean toothpose may be loaded into memory or retrieved from memory. The mean toothpose may also be initialized as a set of matrices, one for each tooth inthe tooth pose. In some embodiments, the mean tooth shapes from block1110 may be placed in their corresponding mean tooth pose at block 1130before the shapes of the teeth are adjusted at block 1120. In otherwords, the order of block 1120 and block 1130 may be swapped.

At block 1140, the arch is scaled such that it is generated with toothdimensions according to the patient's tooth and arch size. In someembodiments, an arch scaling factor Φ is based on the particular toothand arch of a particular patient, as discussed above. In someembodiments, at block 1140 the arch is adjusted such that the scaledarch matches the size of a patient's arch, as determined, for example,by size of the teeth and arch in the patient's scanned arch, orotherwise, as discussed herein. In some embodiments, the scaled arch maybe rendered, for example, on a screen for viewing by a patient or dentalprofessional. In some embodiments, the scaled arch may be stored inmemory. The scaled arch may also be stored as a set of matrices, one foreach tooth in the tooth pose. In some embodiments, the mean tooth shapesfrom block 1110 may be placed in their corresponding scaled position andsize at block 1140 before the shapes of the teeth are adjusted at block1120. In other words, the order of block 1120 and blocks 1130, and 1140may be swapped such that block 1120 takes place after blocks 1130 and1140.

Prior to rendering at block 1150 simulated treatment or viewingcustomizations, such as gingiva line adjustment, jaw position, missingtooth insertion, or broken tooth repair, can be applied to the 3D model.Simulated treatment or viewing customizations may be applied andadjusted as described at block 640. Simulated treatment or viewingcustomizations can be applied before or after arch scaling (e.g., beforeor after block 1140).

At block 1150, the 3D model of the patient's teeth determined at block1140 is rendered for viewing or the final 3D model is stored for lateron-screen rendering. The 3D model can be rendered for viewing or storedfor later on-screen rendering with or without the simulated treatment orviewing customizations.

At block 1160, textures are applied to the 3D model of the patient'steeth in an estimated and/or intended final position. For example,textures determined based on a 2D image of the patient's teeth, forexample, according to process 900 in FIG. 9, may be applied to the 3Dmodel. In some embodiments, the texture application process of block1160 may occur during or as part of the rendering process of block 1150.Textures can be applied to a mask region, where the mask region maydefine a region corresponding to an oral feature, for example, lips,gingiva, or teeth. The mask region or a portion thereof may bedetermined, for example, by simulating the gingiva as shown anddescribed with respect to method 2001 in FIG. 20. During textureapplication, either at block 1160 or during the rendering process ofblock 1150, textures model can be adjusted to simulate treatment or forcustomizable viewing. Simulated treatment or viewing customizations,including color adjustments, e.g., to simulate tooth whitening, can beapplied either to the 3D model or may occur as part of the renderingprocess. Alternatively, color adjustment can be performed at describedat block 1060. Simulated treatment and viewing customizations, coloradjustment can be adjusted by the user.

Optionally, step 1110 can be replaced by a parameterized search of thetreatment plan datastore to identify a matched template model based onthe patient's tooth shape. Steps 1120 through 1160 can subsequently beimplemented as described using the template arch and tooth shape inplace of the mean arch or tooth shape to determine ideal tooth poseinstead of mean tooth pose.

With attention to FIG. 20, a method 2001 of simulating a gingiva lineusing a combination of patient-specific and simulated input parametersis depicted. At block 2011, one or more 2D image(s) of a patient arecaptured or retrieved. The 2D images may be captured as described atblock 110 of FIG. 1. One or more 2D image(s) of a patient are captured.The 2D image may depict the mouth of the patient, for example, in asmiling position. The 2D image may also be obtained as part of theprocesses described at block 610 of FIG. 6, block 1020 of FIG. 10A, orblock 1015 of FIG. 10B. For example, the 2D image captured at block 1020may be retrieved at block 2011 for use in generating gingiva for use inthe 2D image generated at block 1050 or block 1060.

At block 2021, the edges of oral features of the patient, such as theteeth, gingiva, and lips, are determined from the 2D image of thepatient's mouth. Edges of the oral features, for example, the teeth,lips, and gingiva, may be identified as described at block 620 of FIG.6. Facial landmarks from the 2D image of the patient, may be used toidentify the mouth opening, within which oral features, including theteeth and gingiva are located.

An initial determination of the edges of the oral features may beidentified by a machine learning algorithm. Initial tooth contours areextracted from the image and may have a brightness or other scaleapplied to them. The initial contours may then undergo binarization tochange the pixels of the image to a binary scale. The binarized toothcontours are thinned to, for example, a single pixel width, forming athinned tooth contour. The initial determination of the patient's lipsmay be based on facial landmarks, such as the lip landmarks determinedaccording to a machine learning algorithm, such as a convoluted neuralnetwork. As with the tooth contours, the lip contours have a brightnessor other scale applied to them. The lip contours are binarized, thinned,and contoured, as with the initial tooth contours. The binarized andthinned tooth and lip contours may define the edges of the oralfeatures. The edges may be determined for example, as described abovewith respect to FIG. 6.

At block 2031, patient-specific input parameters for gingiva aredetermined. The input parameters may include the patient-specific inputparameters shown in FIG. 21A. In some embodiments, the patient-specificinput parameters may be determined by the shape and scale of one or moreof the oral features of the captured 2D image of the patient. Thepatient-specific input parameters may be determined based on the edgesof the oral features, for example, as illustrated in 2100 of FIG. 21B.The edges of the oral features may include, for example, edges of lips2110, edges teeth 2120, and edges of gingiva 2130, determined at block2021. The patient-specific input parameters may include a patient toothheight, which may be a distance between an incisal edge 2120 of a toothand the edge of the gingiva 2130 or gingiva line and may be determinedfor each tooth of a patient's arch. The patient tooth height may bedenoted by h_(g), as shown, for example, in FIG. 21A. In someembodiments, the patient tooth height may be a maximal distance betweenthe incisal edge 2120 of the tooth and the edge of the gingiva 2130. Insome embodiments, the maximal distance may be measured as the distancebetween the incisal edge 2120 of the tooth and the edge of the gingiva2130 at the gingival zenith.

In some embodiments, the patient-specific input parameters may include agingival tip distance, which may be a distance between the edge of thelip 2110 and the gingiva line 2140. The distance may be determined at aposition between each pair of neighboring teeth of the patient's arch.The gingival tip distance may be denoted by h_(b), as shown, forexample, in FIG. 21A. In some embodiments, the patient tooth height maybe a maximal distance between the edge of the lip 2110 and the edge ofthe gingiva 2140. The maximal may be measured as the distance betweenthe lip and the edge of the gingiva 2140 at the gingival nadir. Thepatient-specific input parameters may include a gingival height, whichmay be a distance between the edge of the lip 2110, and the gingiva line2130, determined for each tooth of a patient's arch, the gingival heightmay be denoted by h_(t), as shown, for example, in FIG. 21A. In someembodiments, the gingival height may be a minimal distance between theedge of the lip 2110 and the gingival zenith 2130. The patient-specificinput parameters may include a tooth width, determined for each tooth ofthe patient's arch. The tooth width may be denoted by w, as shown, forexample, in FIG. 21A. In some embodiments, the tooth width may be awidth of a tooth at the incisal edge 2120. In some embodiments, thetooth width may be a width of a tooth at the maximal tooth width.

At block 2041, simulated parameters for gingiva are determined. Thesimulated parameters may be derived based on the patient-specific inputparameters and may be used in generating the simulated gingiva. In someembodiments, the parameters of gingiva tip parameter, denoted by h,visible tooth height, denoted by g, and tooth curvature, denoted by R,may be a convolution of patient-specific input parameters, as shown inFIG. 22. For example, they may be a convolution of patient-specificparameters and simulated input parameters or ideal aesthetic parameters.An ‘ideal tooth,’ may include a representation of a tooth or parametersof a taken from a model arch, such as those of historical cases and/orthose representing idealized arch forms. An ideal tooth may becharacterized by various parameters, including, without limitation: agingiva tip parameter, an ideal tooth height, a tooth thickness, adistance from the gingiva line to the lip, or a visible tooth height.The patients-specific parameters may be, for example, the parameters ofthe patient tooth height, denoted by h_(g), the gingiva tip distance,denoted by h_(b), the gingival height, denoted by h_(t), and the toothwidth, denoted by w, discussed above. The simulated input parameters orideal aesthetic parameters may be, for example, the parameters of idealtooth height, denoted by H_(g), customizable tooth height scalingparameters denoted by c₁ and c₂, a customizable gingiva tip scalingparameter, denoted by s, and a tooth shape scaling parameter, denoted bym. The visible tooth height, denoted by g, may be a weighted average ofthe visible tooth height of a patient's teeth determined based on the 2Dimage, denoted by h_(g), and an ideal tooth height, denoted by H_(g).The visible tooth height, denoted by g, may be determined based on theweighted average, for example, g=c₁H_(g)c₂h_(g), where c₁ and c₂ arecustomizable tooth height scaling parameters and where c₁+c₂=1. Forexample, c₁ may be 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, or 0.9. Forexample, c₁ may be from 0.1 to 0.5, 0.2 to 0.6, 0.3 to 0.7, 0.4 to 0.8,or 0.5 to 0.9. In some embodiments, c₁ is adjusted by a user, such as adental professional or patient.

In some embodiments, the visible tooth height, denoted by g, representsthe visible tooth height of a patient's teeth determined based on the 2Dimage, denoted by h_(g). A gingiva tip parameter for each tooth, h, maybe determined based on patient-specific input parameters, for example,where h=h_(b)−h_(t). The gingiva tip parameter describes the shape ofthe gingiva surrounding a tooth. A gingiva tip parameter of 0 describesa gingiva line having a straight line across a tooth, and positivegingiva tip parameter describes a gingiva line having an curved shapewhere the distance between the lip and the gingiva line is greaterbetween two teeth than at the midline of a tooth. In some embodiments,the gingiva tip parameter, h, may be determined based the differencebetween the patient-specific tooth height, denoted by h_(g), the idealtooth height, denoted by H_(g), and a customizable gingiva tip scalingparameter, denoted by s, for example, whereh=s(H_(g)−h_(g))(h_(b)−h_(t)) with s determined by the differencebetween Hg and hg. For example when the difference is large, it islikely to have long squared teeth, s then takes a larger value; when thedifference is small, s takes a relatively small value. In someembodiments, s may be adjusted based on the difference of ideal toothlength H_(g) and the patient's tooth length h_(g). For example, s may be0.8, 0.9, 1.0, 1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, or 2.0. Forexample, s may be varied from 0.8 to 1.0, 0.9 to 1.1, 1.0 to 1.2, 1.1 to1.3, 1.2 to 1.4, 1.3 to 1.5, 1.4 to 1.6, 1.5 to 1.7, 1.6 to 1.8, 1.7 to1.9, 1.8 to 2.0, or 0.8 to 2.0. In some embodiments, s is adjusted by auser, such as a dental professional or patient.

A maximum threshold for the gingiva tip parameter, denoted by h, foreach tooth may be set as a fraction of patient tooth height, denoted byh_(g). For example, the gingiva tip parameter, h, may be set so as notto exceed 0.2h_(g), 0.3h_(g), 0.4h_(g), 0.5h_(g), 0.6h_(g), 0.7h_(g),0.75h_(g), 0.8h_(g), 0.85h_(g), or 0.9h_(g). The maximum threshold forthe gingiva tip parameter, h, may be set as a fraction of visible toothheight, denoted by g. For example, h may not exceed 0.2 g, 0.3 g, 0.4 g,0.5 g, 0.6 g, 0.7 g, 0.75 g, 0.8 g, 0.85 g, or 0.9 g.

The tooth curvature parameter, denoted by R, may be a function of thepatient-specific gingiva width parameter, denoted by h_(t), tooththickness, denoted by T, and a tooth shape scaling parameter, denoted bym. In some embodiments, the tooth curvature parameter, R, may bedetermined by R=mh_(t)T. In some embodiments, the tooth thickness,denoted by T, may be determined based on the parameterized 3D model ofthe patient's teeth, for example, where T is the distance between alingual surface and a buccal surface of a tooth at the midpoint of thegingiva line. In some embodiments, m is adjusted based on the patient'stooth length, denoted by h_(g). For example, m may be 0.1, 0.2, 0.3,0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0. For example, m may be varied from 0.1to 0.3, 0.2 to 0.4, 0.3 to 0.5, 0.4 to 0.6, 0.5 to 0.7, 0.6 to 0.8, 0.7to 0.9, or 0.8 to 1.0. In some embodiments, m is adjusted by a user,such as a dental professional or patient.

In some embodiments, the input parameters, for example the parametersdenoted by h, g, and R, are used to determine a parameterized gingivaline. The patient-specific input parameters, the convolution of thepatient-specific input parameters, and the simulated input parameters,or any combination thereof may be used to determine the gingiva line. Insome embodiments, determining the parameterized gingiva line comprisesapproximating the parametric 3D model of a tooth of the patient's teethas a cylinder. The cylinder may be cut cross-sectionally by a tiltedplane, wherein the orientation and location of the tilted plane isdetermined based on the input parameters such that the line ofintersection between the tilted plane and the cylinder passes throughthe coordinates defined by h, g, and R. The line of intersection betweenthe tilted plane and the cylinder may be used to define the edge of thegingiva. In some embodiments, determining the parameterized gingiva linecomprises cutting the 3D model of the patient's teeth cross-sectionallyby a tilted plane, wherein the orientation and location of the tiltedplane is determined based on the input parameters such that the line ofintersection between the tilted plane and the cylinder passes throughthe coordinates defined by h, g, and R. The line of intersection betweenthe tilted plane and the 3D model of the patient's teeth may be used todefine the edge of the gingiva. Points on the gingival side of the lineof intersection between the tilted plane and the cylinder or the 3Dmodel of the patient's teeth are rendered as gingiva, while points of onthe incisal side of the line of intersection between the tilted planeand the cylinder are rendered as teeth. In some embodiments, the pointsare a 3D point cloud of points. In some embodiments, points are pixelsof a 2D image.

At block 2051, the gum line is optionally leveled. The gum line, whichintersects corresponding coordinates of the gingiva line of each tooth,may be adjusted by fitting to a second degree polynomial as shown inFIG. 23. The gum line may be adjusted such that correspondingcoordinates of the gingiva line are positioned along the line defined bythe second degree polynomial fit. In some embodiments, the correspondingcoordinates of each tooth are the gingiva curve end points, or gingivazenith 2130 in FIG. 21B, the point on the gingiva line of a tooth atwhich the distance between the gingiva line and the lip is the shortest.In some embodiments, the corresponding coordinates of each tooth are thegingiva curve tip points, or gingiva nadir 2140 in FIG. 21B, the pointon the gingiva line between two teeth at which the distance between thegingiva line and the lip is the longest.

The gingiva line determined based on the patient-specific and simulatedinput parameters as described in method 2001 and, optionally, leveled asdescribed in FIG. 23, may be used to render the gingiva in the final 2Dimage of the patient's teeth either in the initial position or in thefinal position following simulated treatment as described at block 140of FIG. 1, block 650 of FIG. 6, block 1060 of FIG. 10A, or block 1055 ofFIG. 10B.

The result of the parameterized gingiva line simulation using acombination of the patient-specific parameters, simulated parameters,and ideal aesthetic parameters, illustrated in FIG. 20 and, optionally,gum line leveling, illustrated in FIG. 23, is a gingiva line in therendered 2D image that more closely matches the initial captured 2Dimage and produces a more aesthetically correct simulated 2D image thana parameterized gingiva line defined entirely by patient-specific inputparameters. This improvement may be more apparent in cases where the lipcovers part of one or more teeth, one or more teeth are chipped ordamaged, one or more teeth are crooked, or the patient's width-heighttooth ratio deviates substantially from an ideal width-height toothratio. Additionally, the parameterized gingiva line simulation is lesscomputationally intensive than generating a 3D gum model using a 3D meshtemplate.

At block 2061, the simulated position of the parameterized gingiva lineand 3D tooth model in either the initial position or the final matchedposition are rendered and inserted into the 2D image of the patient.Rendering of the 2D image is described in more detail at block 1060 ofFIG. 10A. The 3D rendering of the teeth may be placed in the mouth ofthe patient as defined by the lip contours. An image of theparameterized gingiva line identified in method 2001 of FIG. 20 may bealso rendered in the 2D image.

In some embodiments, neural networks, such as generative adversarialnetworks or conditional generative adversarial networks may be used tointegrate a 3D model of teeth in a final position with a facial image ofa patient and match the colors, tones, shading, and other aspects of the3D model with a facial photo. The neural network is trained using facialimages. In some embodiments, the facial images may include images ofpeople's faces having a social smiles. In some embodiments, the facialimages may include facial images of patient's teeth before orthodontictreatment. During training, patient's teeth and their contours may beidentified. For example, each tooth may be identified by type (e.g.,upper left central incisor, lower right canine). Other aspects andfeatures of the image may also be identified during training, such asthe location and color of the gingiva, the color of the teeth, therelative brightness of the surfaces within the mouth, and others.

Referencing FIG. 24, after training, the neural network receives inputsat block 4804 for use in generating a realistic rendering of thepatient's teeth in a clinical final position. In some embodiments, theinputs may include one or more of an image of a rendering of a 3D modelof the patient's teeth in a clinical final position or a 3D renderedmodel of the patients teeth in the clinical final position 4808, theclinical final position determined, for example, according to anorthodontic treatment plan, the parameterized gingiva line, a blurredinitial image of the patient's teeth 4810, and a color coded image 4812of the 3D model of the patient's teeth in the clinical final position.

The image of a rendering of a 3D model of the patient's teeth in aclinical final position or the 3D rendered model of the patient's teethin the clinical final position 4808 may be determined based on theclinical orthodontic treatment plan for moving the patient's teeth fromthe initial position towards the final position, as described above. Theimage or rendering 4808 may be generated based on the imagingperspectives. For example, one or more of the imaging distance, thefocal length of the imaging system, and the size of the patient's teethin the initial facial image may be used to generate the image orrendering.

The blurred image of the patient's teeth 4810 may be generated using oneor more blur algorithms, such as a Gaussian blur algorithm. In someembodiments, the Gaussian blur may have a large radius, for example, aradius of at least 5, 10, 20, 40, or 50 pixels. In some embodiments, theblur is sufficiently large such that the tooth structure is notdiscernable in the blurred image.

The coded model of the patient's teeth 4812 may be a red-green-blue(RGB) color coded image of a model of the patient's teeth, with eachcolor channel corresponding to a different quality or feature of themodel. For example, the green color channel, which may be an 8-bit colorchannel indicates the brightness of the blurred image 4810 on a scale of0 to 255 as, for example, overlaid on the 3D model.

The red color channel may be used to differentiate each tooth and thegingiva from each other. In such an embodiment, the gingiva 4813 mayhave a red channel value of 1, the left upper central incisor 4814 mayhave a red value of 2, the right lower canine may have a red channel of3, the portions of the model that are not teeth or gingiva might have ared channel value of 0, and so on, so that the red channel value of eachpixel identifies the dental anatomy associated with the pixel.

The blue color channel may be used to identify the angle of the teethand/or gingiva with respect to the facial plane. For example, at eachpixel location the angle normal to the surface of the dental structure,is determined and a value between 0-255 (for 8-bit color channels) isassigned to the pixel. Such information allows the neural network to,for example, model light reflectivity from the dental surfaces.

The neural network then uses the inputs and its training to render arealistic image 4806 of the patient's teeth in a final position. Thisrealistic image is then integrated into the mouth opening of the facialimage and an alpha channel blurring is applied.

FIG. 12 depicts a system 1200 for simulating an estimated and/orintended outcome of an orthodontic treatment, in accordance with someembodiments. In the example of FIG. 12, the system 1200 includes acomputer-readable medium 1210, a dental scanning system 1220, dentaltreatment planning system 1230, a dental treatment simulation system1240, and an image capture system 1250. One or more of the elements ofthe system 1200 may include elements of such as those described withreference to the computer system shown in FIG. 20. One or more elementsof the system 1200 may also include one or more computer-readable mediaincluding instructions that when executed by a processor, for example, aprocessor of any of the systems 1220, 1230, 1240, and 1250 cause therespective system or systems to perform the processes described herein.

The dental scanning system 1220 may include a computer system configuredto capture one or more scans of a patient's dentition. The dentalscanning system 1220 may include a scan engine for capturing 2D imagesof a patient. Such images may include images of the patient's teeth,face, and jaw. The images may also include x-rays or other subsurfaceimages of the patient. The scan engine may also capture 3D datarepresenting the patient's teeth, face, gingiva, or other aspects of thepatient.

The dental scanning system 1220 may also include a 2D imaging system,such as a still or video camera, an x-ray machine, or other 2D imager.In some embodiments, the dental scanning system 1220 also includes a 3Dimager, such as an intraoral scanner or an impression scanner. Thedental scanning system 1220 and associated engines and imagers can beused to capture the historic scan data for use in determining thehistoric mean parameters of a 3D parametric dental model, as describedwith reference to FIGS. 1-11 herein. The dental scanning system 1220 andassociated engines and imagers can be used to capture the 2D images of apatient's face and dentition for use in building a 3D parametric modelof the patient's teeth, as described for example, with reference toFIGS. 1-11 herein.

The dental treatment simulation system 1240 may include a computersystem configured to simulate one or more estimated and/or intendedoutcomes of a dental treatment plan. In some implementations, the dentaltreatment simulation system 1240 obtains photos and/or other 2D imagesof a consumer/patient. The dental treatment simulation system 1240 mayfurther be configured to determine tooth, lip, gingiva, and/or otheredges related to teeth in the 2D image. As noted herein, the dentaltreatment simulation system 1240 may be configured to match tooth and/orarch parameters to tooth, lip, gingiva, and/or other edges. The dentaltreatment simulation system 1240 may also render a 3D tooth model of thepatient's teeth. The dental treatment simulation system 1240 may gatherinformation related to historical and/or idealized arches representingan estimated outcome of treatment. The dental treatment simulationsystem 1240 may, in various implementations, insert, align, etc., the 3Dtooth model with the 2d image of the patient in order to render a 2Dsimulation of an estimated outcome of orthodontic treatment. The dentaltreatment simulation system 1240 may include a photo parameterizationengine which may further include an edge analysis engine, an EM analysisengine, a course tooth alignment engine, and a 3D parameterizationconversion engine. The dental treatment simulation system 1240 may alsoinclude a parametric treatment prediction engine which may furtherinclude a treatment parameterization engine, a scanned toothnormalization engine, and a treatment plan remodeling engine. The dentaltreatment simulation system 1240 and its associated engines may carryout the processes described above with respect to FIGS. 5-9.

The dental treatment planning system 1230 may include a computer systemconfigured to implement treatment plans. The dental treatment planningsystem 1230 may include a rendering engine and interface for visualizingor otherwise displaying the simulated outcome of the dental treatmentplan. For example, the rendering engine may render the visualizations ofthe 3D models described herein, for example, at block 140 of FIG. 1, atblocks 640, 650, and 660 of FIG. 6, process 800, described withreference to FIG. 8, at blocks 910, 920, and 930 of FIG. 9, block 1150of FIG. 10, block 1150 of FIG. 11, and block 2061 of FIG. 20. Therendering engine may render the realistic rendering of the patient'steeth in a clinical final position, described with reference to FIG. 24.The dental treatment planning system 1230 may also determine anorthodontic treatment plan for moving a patient's teeth from an initialposition, for example, based in part on the 2D image of the patient'steeth, to a final position. The dental treatment planning system 1230may be operative to provide for image viewing and manipulation such thatrendered images may be scrollable, pivotable, zoomable, and interactive.The dental treatment planning system 1230 may include graphics renderinghardware, one or more displays, and one or more input devices. Some orall of the dental treatment planning system 1230 may be implemented on apersonal computing device such as a desktop computing device or ahandheld device, such as a mobile phone. In some embodiments, at least aportion of the dental treatment planning system 1230 may be implementedon a scanning system, such as the dental scanning system 1220. The imagecapture system 1250 may include a device configured to obtain an image,including an image of a patient. The image capture system may compriseany type of mobile device (iOS devices, iPhones, iPads, iPods, etc.,Android devices, portable devices, tablets), PCs, cameras (DSLR cameras,film cameras, video cameras, still cameras, etc.). In someimplementations, the image capture system 1250 comprises a set of storedimages, such as images stored on a storage device, a network location, asocial media website, etc.

FIG. 13 shows an example of one or more of the elements of the dentaltreatment simulation system 1240, in accordance with some embodiments.In the example of FIG. 13, the dental treatment simulation system 1240includes a photo gathering engine 1310, a photo datastore 1360, a photoparameterization engine 1340, a case management engine 1350, a referencecase datastore 1370, and a treatment rendering engine 1330.

The photo gathering engine 1310 may implement one or more automatedagents configured to retrieve a selected photo of a patient from thephoto datastore 1360, the image capture system 1250, and/or a scan fromthe dental scanning system 1220. The photo gathering engine may thenprovide the photo or photos to the photo parameterization engine 1340.The photo gathering engine may then provide the photo or photos to thephoto parameterization engine 1340 and/or other modules of the system.

The photo datastore 1360 may include a datastore configured to storephotos of patients, such as their facial photos. In some embodiments,the photos are 2D images that include the mouth of the patient and oneor more images of the face, head, neck, shoulders, torso, or the entirepatient. The 2D image of the patient may include an image of the patientwith their mouth in one or more positions; for example, the patient'smouth may be a smiling position, such as a social smiling position, arepose position with relaxed muscles and lips slightly parted, oranterior retracted open bite or closed bite positions. In someembodiments the image of the patient is taken with an image capturesystem. The image may be captured with a lens at a predetermined focallength and at a distance from the patient. The image may be capturedremotely and then received for processing. The image may be a series ofimages of video captured from one or more perspectives. For example, theimages may include one or more of a frontal facial image and a profileimage, including one or more three-quarter profile images and fullprofile images.

The photo parameterization engine 1340 may implement one or moreautomated agents configured to build a parametric 3D model based on the2D image of the patient from the photo datastore 1360 and the parametricmodel and mean data from the parametric treatment prediction engine. Theedge analysis engine 1342 may implement one or more automated agentsconfigured to determine the edges of the teeth, lips and gingiva withinthe photo of the patient. For example as described with respect to FIG.6. In particular, an initial determination of the patient's lips (forexample, the inner edges of the lips that define the mount opening),teeth, and gingiva contours may be identified by a machine learningalgorithm, such as a convoluted neural network.

Then, the initial contours may be extracted from the image. The pixelsthat denote the contours may then undergo binarization. The binarizedtooth contours are thinned, whereby the contour's thickness is reducedto, for example, a single pixel in width, forming a thinned contour.

Using the contours, either thinned or otherwise, parameterized 3D teethand arch models are matched to each of the patient's teeth that aredepicted in the 2D image of the patient. The matching is based on thecontours, determined above.

The coarse alignment engine 1344 may implement one or more automatedagents configured to receive the 2D image and/or the associated edgesand performs a coarse alignment between the identified edges and themean tooth models, for example, as described with reference to FIG. 7A,wherein the respective centers of the teeth in the 2D image and thesilhouette of the parametric tooth may be aligned before the 2D imageand/or associated edges, along with the silhouettes and their associatedtooth parameters are sent to the EM engine 1346.

The expectation maximization (EM) engine 1346 may implement one or moreautomated agents configured to perform an expectation-maximizationanalysis between the edges of the 2D image and the parameters of the 3Dparametric model, for example as described with respect to FIG. 7A, andin particular, blocks 720, 730, and 740.

Once the EM analysis engine 1346 completes the matching of theparametric 3D model to the 2D image of the patient, the parametricresults are sent to the 3D parameter conversion engine 1348, which mayconvert the output principle component analysis of the EM analysisengine to case-specific parameters that define a 3D model of thepatient's teeth according to a parametric model for use by theparametric treatment prediction engine 1350, as described above.

The gingiva parameterization engine 1345 may implement one or moreautomated agents configured to build a gingiva line model based onpatient-specific input parameters and, optionally, simulated inputparameters, as described with respect to FIG. 20, and in particular,blocks 2031 and 2041. The gingiva line model is generated using inputfrom the initial 2D image of the patient's teeth, the 3D parametricmodel of the patient's teeth, and, optionally, ideal tooth shapeparameters. The gingiva line model may then be rendered into the 2Dimage using the treatment rendering engine, as described at block 1330.

The gum line leveling engine 1347 may implement one or more automatedagents configured to level the gum line based on gingiva line parametersgenerated at block 1345, the gingiva parameterization engine. Gum lineleveling is, optionally, performed as described with respect to FIG. 20,and in particular, block 2051. The leveled gum line may then be renderedinto the 2D image using the treatment rendering engine, as described atblock 1330.

The case management engine 1350 may include a case parameterizationengine 1352, a scanned tooth normalization engine 1354, and a treatmentplan simulation engine 1356. The case management engine 1350 mayimplement one or more automated agents configured to define theparametric model used to represent the teeth and arch of patients,determine the mean data to parametrically represent the mean positionsof a patient's teeth and arch based on the treatment plans retrievedfrom the treatment plan datastore 1370, and simulate the treatment of aparticular patient's teeth based on the parametric model, the mean toothpositions, and a patient's particular teeth.

The case parameterization engine 1352 may implement one or moreautomated agents configured to define the parametric 3D model for use inmodeling teeth. For example, the parametric 3D model may be defined asdescribed above with reference to eq. (2). More than just defining theequation, treatment parameterization engine 1352 may also define theschema for the parameters. For example, the T_(τ), which representstooth positions, may be defined as a 4*4 matrix including tooth positionand rotation in 3 axis. Similarly, a_(τ) ^(i), which represents toothshape, may be defined as an 2500*3 matrix, where each vertex of thesphere is mapped to a location on the surface of the tooth model, eachlocation of a vertex being x, y, and z special locations. For lower orhigher definition models, fewer or greater vertices may be used. Furtherdiscussion of the parameters defined by the treatment parameterizationengine 1352 are descried elsewhere herein, for example with respect toFIGS. 2-5.

The scanned tooth normalization engine 1354 may implement one or moreautomated agents configured to parameterize the scanned teeth of a setof treatment plans gathered from the treatment plan datastore 1370 andthen determines the mean set of general generic parameters for theparametric model. The scanned tooth normalization engine 1354 may carryout process 350, as described with reference to FIG. 3B. For example,the scanned tooth normalization engine 1354 may align the arches withinthe historic cases retrieved from the datastore 1370. Each arch in thedatastore is aligned at three locations: between the central incisorsand at each distal end of the left and right side of each arch. Eachtooth's position and orientation is then determined based on itslocation and orientation with reference to the alignment points.

The scanned tooth normalization engine 1354 may implement one or moreautomated agents configured to determine the distributions of theparameters among the cases. For example, each historic arch may berescaled to determine Φ, then each arch is averaged to determine T_(τ)^(C), then the individual tooth local transformations are determined andcompared to T_(τ) ^(C) to determine T_(τ), then after aligning eachtooth β_(τ) is determined.

From this data, each tooth's relative shape, location, and rotation aredetermined in order to build the distribution for each case-specificparameter. For example the distribution of the a_(τ) ^(i), thecoefficients for the principal components, for the surface model of theeach respective tooth in all of the retrieved models is determined.

The location and orientation of each tooth is averaged to determine amean location for each tooth and the orientation of each tooth isaveraged to determine a mean orientation for each tooth. The meanlocation and orientation are used to determine T_(τ) ^(C).

In some embodiments, the scanned teeth of a set of treatment plansgathered from the treatment plan datastore can be used to generate atemplate datastore of parameterized template models. The scanned toothnormalization engine 1354 may align the arches within the historic casesretrieved from the reference case datastore 1370. Each arch in thereference case datastore is aligned at three locations: between thecentral incisors and at each distal end of the left and right side ofeach arch. Each tooth's position and orientation is then determinedbased on its location and orientation with reference to the alignmentpoints. The parameters of the parameterized template models in thedatastore are searchable. The datastore can be used to match patient 3Dmodels (e.g., generated from 2D models or from 3D scans) to the closesttemplate in the datastore based on tooth shape and arch shape toidentify a matched template model. The closest matched template modelcan be used to determine the final tooth positions and orientations forthe patient's teeth.

The treatment plan simulation engine 1356 may implement one or moreautomated agents configured to use the parametric model defined by thetreatment parameterization engine 1352, the generic parameters from thescanned tooth normalization engine and the case-specific parameters fromthe photo parameterization engine 1340 to simulate an estimated and/orintended outcome of a dental treatment plan of a specific patient'steeth, for example, as described with respect to FIGS. 10 and 11.

The treatment plan simulation engine 1356 retrieves or otherwisedetermines the mean shape, S_(τ) ^(C). In some embodiments, the meanshape may be retrieved from scanned tooth normalization engine 1354. Insome embodiments, the mean shape may be loaded into memory or retrievedfrom memory. The mean shape may also be initialized as a set ofmatrices, one for each tooth.

The treatment plan simulation engine 1356 performed a principalcomponent analysis shape adjustment on the mean shape of the teeth,S_(τ) ^(C). For each tooth in the model, the case specific coefficientsas determined from the photo parameterization engine 1340 for theprincipal component, a_(τ) ^(i), are applied to the principal component,B_(τ) ^(i). After the shape adjustment is completed, in someembodiments, the adjusted shape may be rendered, for example, on ascreen for viewing by a patient or dental professional. In someembodiments, the adjusted shape may be stored into memory. The adjustedshape may also be stored as a set of matrices, one for each tooth.

The treatment plan simulation engine 1356 determines the mean toothpose. In some embodiments, the mean tooth pose may be based on the meantooth pose of a set of scanned arches, as discussed above. In someembodiments, each adjusted tooth is placed in its corresponding meanlocation and orientation, as determined by the mean arch. In someembodiments, the mean tooth pose may be rendered, for example, on ascreen for viewing by a patient or dental professional. In someembodiments, the mean tooth pose may be loaded into memory or retrievedfrom memory. The mean tooth pose may also be initialized as a set ofmatrices, one for each tooth in the tooth pose.

In some embodiments, the treatment plan remodeling engine 1356 uses thetooth position and orientation coordinates from the ideal parameterizeddatastore match and applies the shape and textures of the patient's 3Dmodel generated from the 2D image to the tooth positions andorientations to generate an arch.

The treatment plan simulation engine 1356 scales the arch such that itis generated according to the patient's tooth and arch size. In someembodiments, the arch scaling factor Φ is based on the particular toothand arch of a particular patient, as discussed above. In someembodiments, the arch is adjusted such that the scaled arch matches thesize of a patient's arch, as determined, for example, by size of theteeth and arch in the patient's scanned arch, or otherwise, as discussedherein. In some embodiments, the scaled arch may be rendered, forexample, on a screen for viewing by a patient or dental professional. Insome embodiments, the scaled arch may be stored in memory. The scaledarch may also be stored as a set of matrices, one for each tooth in thetooth pose. The scaled arch may be sent to the treatment renderingengine 1330.

The treatment rendering engine 1330 renders the teeth and 2D image ofthe patient in a final position, as determined, for example, by theparametric treatment prediction engine 1350 and, in particular, thetreatment plan simulation engine 1356 therein. The treatment renderingengine 1330 may carry out some of the processes described with referenceto FIGS. 9, 10, and 11. In particular, the rendering processes descriedwith reference to FIGS. 9, 10, 11, and elsewhere herein.

FIG. 14 illustrates an exemplary tooth repositioning appliance oraligner 1500 that can be worn by a patient in order to achieve anincremental repositioning of individual teeth 1502 in the jaw. Theappliance can include a shell (e.g., a continuous polymeric shell or asegmented shell) having teeth-receiving cavities that receive andresiliently reposition the teeth. An appliance or portion(s) thereof maybe indirectly fabricated using a physical model of teeth. For example,an appliance (e.g., polymeric appliance) can be formed using a physicalmodel of teeth and a sheet of suitable layers of polymeric material. Thephysical model (e.g., physical mold) of teeth can be formed through avariety of techniques, including 3D printing. The appliance can beformed by thermoforming the appliance over the physical model. In someembodiments, a physical appliance is directly fabricated, e.g., usingadditive manufacturing techniques, from a digital model of an appliance.In some embodiments, the physical appliance may be created through avariety of direct formation techniques, such as 3D printing. Anappliance can fit over all teeth present in an upper or lower jaw, orless than all of the teeth. The appliance can be designed specificallyto accommodate the teeth of the patient (e.g., the topography of thetooth-receiving cavities matches the topography of the patient's teeth),and may be fabricated based on positive or negative models of thepatient's teeth generated by impression, scanning, and the like.Alternatively, the appliance can be a generic appliance configured toreceive the teeth, but not necessarily shaped to match the topography ofthe patient's teeth. In some cases, only certain teeth received by anappliance will be repositioned by the appliance while other teeth canprovide a base or anchor region for holding the appliance in place as itapplies force against the tooth or teeth targeted for repositioning. Insome cases, some or most, and even all, of the teeth will berepositioned at some point during treatment. Teeth that are moved canalso serve as a base or anchor for holding the appliance as it is wornby the patient. In some embodiments, no wires or other means will beprovided for holding an appliance in place over the teeth. In somecases, however, it may be desirable or necessary to provide individualattachments or other anchoring elements 1504 on teeth 1502 withcorresponding receptacles or apertures 1506 in the appliance 1500 sothat the appliance can apply a selected force on the tooth. Exemplaryappliances, including those utilized in the Invisalign® System, aredescribed in numerous patents and patent applications assigned to AlignTechnology, Inc. including, for example, in U.S. Pat. Nos. 6,450,807,and 5,975,893, as well as on the company's website, which is accessibleon the World Wide Web (see, e.g., the url “invisalign.com”). Examples oftooth-mounted attachments suitable for use with orthodontic appliancesare also described in patents and patent applications assigned to AlignTechnology, Inc., including, for example, U.S. Pat. Nos. 6,309,215 and6,830,450.

Optionally, in cases involving more complex movements or treatmentplans, it may be beneficial to utilize auxiliary components (e.g.,features, accessories, structures, devices, components, and the like) inconjunction with an orthodontic appliance. Examples of such accessoriesinclude but are not limited to elastics, wires, springs, bars, archexpanders, palatal expanders, twin blocks, occlusal blocks, bite ramps,mandibular advancement splints, bite plates, pontics, hooks, brackets,headgear tubes, springs, bumper tubes, palatal bars, frameworks,pin-and-tube apparatuses, buccal shields, buccinator bows, wire shields,lingual flanges and pads, lip pads or bumpers, protrusions, divots, andthe like. In some embodiments, the appliances, systems and methodsdescribed herein include improved orthodontic appliances with integrallyformed features that are shaped to couple to such auxiliary components,or that replace such auxiliary components.

FIG. 15 illustrates a tooth repositioning system 1510 including aplurality of appliances 1512, 1514, 1516. Any of the appliancesdescribed herein can be designed and/or provided as part of a set of aplurality of appliances used in a tooth repositioning system. Eachappliance may be configured so a tooth-receiving cavity has a geometrycorresponding to an intermediate or final tooth arrangement intended forthe appliance. The patient's teeth can be progressively repositionedfrom an initial tooth arrangement towards a target tooth arrangement byplacing a series of incremental position adjustment appliances over thepatient's teeth. For example, the tooth repositioning system 1510 caninclude a first appliance 1512 corresponding to an initial tootharrangement, one or more intermediate appliances 1514 corresponding toone or more intermediate arrangements, and a final appliance 1516corresponding to a target arrangement. A target tooth arrangement can bea planned final tooth arrangement selected for the patient's teeth atthe end of all planned orthodontic treatment. Alternatively, a targetarrangement can be one of some intermediate arrangements for thepatient's teeth during the course of orthodontic treatment, which mayinclude various different treatment scenarios, including, but notlimited to, instances where surgery is recommended, where interproximalreduction (IPR) is appropriate, where a progress check is scheduled,where anchor placement is best, where palatal expansion is desirable,where restorative dentistry is involved (e.g., inlays, onlays, crowns,bridges, implants, veneers, and the like), etc. As such, it isunderstood that a target tooth arrangement can be any planned resultingarrangement for the patient's teeth that follows one or more incrementalrepositioning stages. Likewise, an initial tooth arrangement can be anyinitial arrangement for the patient's teeth that is followed by one ormore incremental repositioning stages.

FIG. 16 illustrates a method 1550 of orthodontic treatment using aplurality of appliances, in accordance with embodiments. The method 1550can be practiced using any of the appliances or appliance sets describedherein. In block 1560, a first orthodontic appliance is applied to apatient's teeth in order to reposition the teeth from a first tootharrangement to a second tooth arrangement. In block 1570, a secondorthodontic appliance is applied to the patient's teeth in order toreposition the teeth from the second tooth arrangement to a third tootharrangement. The method 1550 can be repeated as necessary using anysuitable number and combination of sequential appliances in order toincrementally reposition the patient's teeth from an initial arrangementto a target arrangement. The appliances can be generated all at the samestage or in sets or batches (at the beginning of a stage of thetreatment, at an intermediate stage of treatment, etc.), or theappliances can be fabricated one at a time, and the patient can weareach appliance until the pressure of each appliance on the teeth can nolonger be felt or until the maximum amount of expressed tooth movementfor that given stage has been achieved. A plurality of differentappliances (e.g., a set) can be designed and even fabricated prior tothe patient wearing any appliance of the plurality. After wearing anappliance for an appropriate period of time, the patient can replace thecurrent appliance with the next appliance in the series until no moreappliances remain. The appliances are generally not affixed to the teethand the patient may place and replace the appliances at any time duringthe procedure (e.g., patient-removable appliances). The final applianceor several appliances in the series may have a geometry or geometriesselected to overcorrect the tooth arrangement. For instance, one or moreappliances may have a geometry that would (if fully achieved) moveindividual teeth beyond the tooth arrangement that has been selected asthe “final.” Such over-correction may be desirable in order to offsetpotential relapse after the repositioning method has been terminated(e.g., permit movement of individual teeth back toward theirpre-corrected positions). Over-correction may also be beneficial tospeed the rate of correction (e.g., an appliance with a geometry that ispositioned beyond a desired intermediate or final position may shift theindividual teeth toward the position at a greater rate). In such cases,the use of an appliance can be terminated before the teeth reach thepositions defined by the appliance. Furthermore, over-correction may bedeliberately applied in order to compensate for any inaccuracies orlimitations of the appliance.

The various embodiments of the orthodontic appliances presented hereincan be fabricated in a wide variety of ways. In some embodiments, theorthodontic appliances herein (or portions thereof) can be producedusing direct fabrication, such as additive manufacturing techniques(also referred to herein as “3D printing) or subtractive manufacturingtechniques (e.g., milling). In some embodiments, direct fabricationinvolves forming an object (e.g., an orthodontic appliance or a portionthereof) without using a physical template (e.g., mold, mask etc.) todefine the object geometry.

In some embodiments, the orthodontic appliances herein can be fabricatedusing a combination of direct and indirect fabrication techniques, suchthat different portions of an appliance can be fabricated usingdifferent fabrication techniques and assembled in order to form thefinal appliance. For example, an appliance shell can be formed byindirect fabrication (e.g., thermoforming), and one or more structuresor components as described herein (e.g., auxiliary components, powerarms, etc.) can be added to the shell by direct fabrication (e.g.,printing onto the shell).

The configuration of the orthodontic appliances herein can be determinedaccording to a treatment plan for a patient, e.g., a treatment planinvolving successive administration of a plurality of appliances forincrementally repositioning teeth. Computer-based treatment planningand/or appliance manufacturing methods can be used in order tofacilitate the design and fabrication of appliances. For instance, oneor more of the appliance components described herein can be digitallydesigned and fabricated with the aid of computer-controlledmanufacturing devices (e.g., computer numerical control (CNC) milling,computer-controlled additive manufacturing such as 3D printing, etc.).The computer-based methods presented herein can improve the accuracy,flexibility, and convenience of appliance fabrication.

In some embodiments, computer-based 3D planning/design tools, such asTreat™ software from Align Technology, Inc., may be used to design andfabricate the orthodontic appliances described herein.

FIG. 17 illustrates a method 1800 for designing an orthodontic applianceto be fabricated, in accordance with embodiments. The method 1800 can beapplied to any embodiment of the orthodontic appliances describedherein. Some or all of the operations of the method 200 can be performedby any suitable data processing system or device, e.g., one or moreprocessors configured with suitable instructions.

In block 1810, a movement path to move one or more teeth from an initialarrangement to a target arrangement is determined. The initialarrangement can be determined from a mold or a scan of the patient'steeth or mouth tissue, e.g., using wax bites, direct contact scanning,x-ray imaging, tomographic imaging, sonographic imaging, and othertechniques for obtaining information about the position and structure ofthe teeth, jaws, gums and other orthodontically relevant tissue. Fromthe obtained data, a digital data set can be derived that represents theinitial (e.g., pretreatment) arrangement of the patient's teeth andother tissues. Optionally, the initial digital data set is processed tosegment the tissue constituents from each other. For example, datastructures that digitally represent individual tooth crowns can beproduced. Advantageously, digital models of entire teeth can beproduced, including measured or extrapolated hidden surfaces and rootstructures, as well as surrounding bone and soft tissue.

The target arrangement of the teeth (e.g., a desired and intended endresult of orthodontic treatment) can be received from a clinician in theform of a prescription, can be calculated from basic orthodonticprinciples, and/or can be extrapolated computationally from a clinicalprescription. With a specification of the desired final positions of theteeth and a digital representation of the teeth themselves, the finalposition and surface geometry of each tooth can be specified to form acomplete model of the tooth arrangement at the desired end of treatment.

Having both an initial position and a target position for each tooth, amovement path can be defined for the motion of each tooth. In someembodiments, the movement paths are configured to move the teeth in thequickest fashion with the least amount of round-tripping to bring theteeth from their initial positions to their desired target positions.The tooth paths can optionally be segmented, and the segments can becalculated so that each tooth's motion within a segment stays withinthreshold limits of linear and rotational translation. In this way, theend points of each path segment can constitute a clinically viablerepositioning, and the aggregate of segment end points can constitute aclinically viable sequence of tooth positions, so that moving from onepoint to the next in the sequence does not result in a collision ofteeth.

In block 1820, a force system to produce movement of the one or moreteeth along the movement path is determined. A force system can includeone or more forces and/or one or more torques. Different force systemscan result in different types of tooth movement, such as tipping,translation, rotation, extrusion, intrusion, root movement, etc.Biomechanical principles, modeling techniques, forcecalculation/measurement techniques, and the like, including knowledgeand approaches commonly used in orthodontia, may be used to determinethe appropriate force system to be applied to the tooth to accomplishthe tooth movement. In determining the force system to be applied,sources may be considered including literature, force systems determinedby experimentation or virtual modeling, computer-based modeling,clinical experience, minimization of unwanted forces, etc.

Determination of the force system can be performed in a variety of ways.For example, in some embodiments, the force system is determined on apatient-by-patient basis, e.g., using patient-specific data.Alternatively or in combination, the force system can be determinedbased on a generalized model of tooth movement (e.g., based onexperimentation, modeling, clinical data, etc.), such thatpatient-specific data is not necessarily used. In some embodiments,determination of a force system involves calculating specific forcevalues to be applied to one or more teeth to produce a particularmovement. Alternatively, determination of a force system can beperformed at a high level without calculating specific force values forthe teeth. For instance, block 1820 can involve determining a particulartype of force to be applied (e.g., extrusive force, intrusive force,translational force, rotational force, tipping force, torquing force,etc.) without calculating the specific magnitude and/or direction of theforce.

In block 1830, an appliance geometry and/or material composition for anorthodontic appliance configured to produce the force system isdetermined. The appliance can be any embodiment of the appliancesdiscussed herein, such as an appliance having variable localizedproperties, integrally formed components, and/or power arms.

For example, in some embodiments, the appliance comprises aheterogeneous thickness, a heterogeneous stiffness, or a heterogeneousmaterial composition. In some embodiments, the appliance comprises twoor more of a heterogeneous thickness, a heterogeneous stiffness, or aheterogeneous material composition. In some embodiments, the appliancecomprises a heterogeneous thickness, a heterogeneous stiffness, and aheterogeneous material composition. The heterogeneous thickness,stiffness, and/or material composition can be configured to produce theforce system for moving the teeth, e.g., by preferentially applyingforces at certain locations on the teeth. For example, an appliance withheterogeneous thickness can include thicker portions that apply moreforce on the teeth than thinner portions. As another example, anappliance with heterogeneous stiffness can include stiffer portions thatapply more force on the teeth than more elastic portions. Variations instiffness can be achieved by varying the appliance thickness, materialcomposition, and/or degree of photopolymerization, as described herein.

In some embodiments, determining the appliance geometry and/or materialcomposition comprises determining the geometry and/or materialcomposition of one or more integrally formed components to be directlyfabricated with an appliance shell. The integrally formed component canbe any of the embodiments described herein. The geometry and/or materialcomposition of the integrally formed component(s) can be selected tofacilitate application of the force system onto the patient's teeth. Thematerial composition of the integrally formed component can be the sameas or different from the material composition of the shell.

In some embodiments, determining the appliance geometry comprisesdetermining the geometry for a variable gable bend.

The block 1830 can involve analyzing the desired force system in orderto determine an appliance geometry and material composition that wouldproduce the force system. In some embodiments, the analysis involvesdetermining appliance properties (e.g., stiffness) at one or morelocations that would produce a desired force at the one or morelocations. The analysis can then involve determining an appliancegeometry and material composition at the one or more locations toachieve the specified properties. Determination of the appliancegeometry and material composition can be performed using a treatment orforce application simulation environment. A simulation environment caninclude, e.g., computer modeling systems, biomechanical systems orapparatus, and the like. Optionally, digital models of the applianceand/or teeth can be produced, such as finite element models. The finiteelement models can be created using computer program applicationsoftware available from a variety of vendors. For creating solidgeometry models, computer aided engineering (CAE) or computer aideddesign (CAD) programs can be used, such as the AutoCAD® softwareproducts available from Autodesk, Inc., of San Rafael, Calif. Forcreating finite element models and analyzing them, program products froma number of vendors can be used, including finite element analysispackages from ANSYS, Inc., of Canonsburg, Pa., and SIMULIA (Abaqus)software products from Dassault Systèmes of Waltham, Mass.

Optionally, one or more appliance geometries and material compositionscan be selected for testing or force modeling. As noted above, a desiredtooth movement, as well as a force system required or desired foreliciting the desired tooth movement, can be identified. Using thesimulation environment, a candidate appliance geometry and compositioncan be analyzed or modeled for determination of an actual force systemresulting from use of the candidate appliance. One or more modificationscan optionally be made to a candidate appliance, and force modeling canbe further analyzed as described, e.g., in order to iterativelydetermine an appliance design that produces the desired force system.

Optionally, block 1830 can further involve determining the geometry ofone or more auxiliary components to be used in combination with theorthodontic appliance in order to exert the force system on the one ormore teeth. Such auxiliaries can include one or more of tooth-mountedattachments, elastics, wires, springs, bite blocks, arch expanders,wire-and-bracket appliances, shell appliances, headgear, or any otherorthodontic device or system that can be used in conjunction with theorthodontic appliances herein. The use of such auxiliary components maybe advantageous in situations where it is difficult for the appliancealone to produce the force system. Additionally, auxiliary componentscan be added to the orthodontic appliance in order to provide otherdesired functionalities besides producing the force system, such asmandibular advancement splints to treat sleep apnea, pontics to improveaesthetic appearance, and so on. In some embodiments, the auxiliarycomponents are fabricated and provided separately from the orthodonticappliance. Alternatively, the geometry of the orthodontic appliance canbe modified to include one or more auxiliary components as integrallyformed components.

In block 1840, instructions for fabrication of the orthodontic appliancehaving the appliance geometry and material composition are generated.The instructions can be configured to control a fabrication system ordevice in order to produce the orthodontic appliance with the specifiedappliance geometry and material composition. In some embodiments, theinstructions are configured for manufacturing the orthodontic applianceusing direct fabrication (e.g., stereolithography, selective lasersintering, fused deposition modeling, 3D printing, continuous directfabrication, multi-material direct fabrication, etc.). Optionally, theinstructions can be configured to cause a fabrication machine todirectly fabricate the orthodontic appliance with teeth receivingcavities having variable gable bends, as discussed above and herein. Inalternative embodiments, the instructions can be configured for indirectfabrication of the appliance, e.g., by thermoforming.

Although the above blocks show a method 1800 of designing an orthodonticappliance in accordance with some embodiments, a person of ordinaryskill in the art will recognize some variations based on the teachingdescribed herein. Some of the blocks may comprise sub-blocks. Some ofthe blocks may be repeated as often as desired. One or more blocks ofthe method 200 may be performed with any suitable fabrication system ordevice, such as the embodiments described herein. Some of the blocks maybe optional, and the order of the blocks can be varied as desired. Forinstance, in some embodiments, block 1820 is optional, such that block1830 involves determining the appliance geometry and/or materialcomposition based directly on the tooth movement path rather than basedon the force system.

FIG. 18 illustrates a method 1900 for digitally planning an orthodontictreatment and/or design or fabrication of an appliance, in accordancewith embodiments. The method 1900 can be applied to any of the treatmentprocedures described herein and can be performed by any suitable dataprocessing system.

In block 1910, a digital representation of a patient's teeth isreceived. The digital representation can include surface topography datafor the patient's intraoral cavity (including teeth, gingival tissues,etc.). The surface topography data can be generated by directly scanningthe intraoral cavity, a physical model (positive or negative) of theintraoral cavity, or an impression of the intraoral cavity, using asuitable scanning device (e.g., a handheld scanner, desktop scanner,etc.).

In block 1920, one or more treatment stages are generated based on thedigital representation of the teeth. The treatment stages can beincremental repositioning stages of an orthodontic treatment proceduredesigned to move one or more of the patient's teeth from an initialtooth arrangement to a target arrangement. For example, the treatmentstages can be generated by determining the initial tooth arrangementindicated by the digital representation, determining a target tootharrangement, and determining movement paths of one or more teeth in theinitial arrangement necessary to achieve the target tooth arrangement.The movement path can be optimized based on minimizing the totaldistance moved, preventing collisions between teeth, avoiding toothmovements that are more difficult to achieve, or any other suitablecriteria.

In block 1930, at least one orthodontic appliance is fabricated based onthe generated treatment stages. For example, a set of appliances can befabricated, each shaped according to a tooth arrangement specified byone of the treatment stages, such that the appliances can besequentially worn by the patient to incrementally reposition the teethfrom the initial arrangement to the target arrangement. The applianceset may include one or more of the orthodontic appliances describedherein. The fabrication of the appliance may involve creating a digitalmodel of the appliance to be used as input to a computer-controlledfabrication system. The appliance can be formed using direct fabricationmethods, indirect fabrication methods, or combinations thereof, asdesired.

In some instances, staging of various arrangements or treatment stagesmay not be necessary for design and/or fabrication of an appliance. Asillustrated by the dashed line in FIG. 18, design and/or fabrication ofan orthodontic appliance, and perhaps a particular orthodontictreatment, may include use of a representation of the patient's teeth(e.g., receive a digital representation of the patient's teeth 1910),followed by design and/or fabrication of an orthodontic appliance basedon a representation of the patient's teeth in the arrangementrepresented by the received representation.

Optionally, some or all of the blocks of the method 1900 are performedlocally at the site where the patient is being treated and during asingle patient visit, referred to herein as “chair side manufacturing.”Chair side manufacturing can involve, for example, scanning thepatient's teeth, automatically generating a treatment plan withtreatment stages, and immediately fabricating one or more orthodonticappliance(s) to treat the patient using a chair side direct fabricationmachine, all at the treating professional's office during a singleappointment. In embodiments where a series of appliances are used totreat the patient, the first appliance may be produced chair side forimmediate delivery to the patient, with the remaining appliancesproduced separately (e.g., off site at a lab or central manufacturingfacility) and delivered at a later time (e.g., at a follow upappointment, mailed to the patient). Alternatively, the methods hereincan accommodate production and immediate delivery of the entire seriesof appliances on site during a single visit. Chair side manufacturingcan thus improve the convenience and speed of the treatment procedure byallowing the patient to immediately begin treatment at thepractitioner's office, rather than having to wait for fabrication anddelivery of the appliances at a later date. Additionally, chair sidemanufacturing can provide improved flexibility and efficiency oforthodontic treatment. For instance, in some embodiments, the patient isre-scanned at each appointment to determine the actual positions of theteeth, and the treatment plan is updated accordingly. Subsequently, newappliances can be immediately produced and delivered chair side toaccommodate any changes to or deviations from the treatment plan.

FIG. 19 is a simplified block diagram of a data processing system 2000that may be used in executing methods and processes described herein.The data processing system 2000 typically includes at least oneprocessor 2002 that communicates with one or more peripheral devices viabus subsystem 2004. These peripheral devices typically include a storagesubsystem 2006 (memory subsystem 2008 and file storage subsystem 2014),a set of user interface input and output devices 2018, and an interfaceto outside networks 2016. This interface is shown schematically as“Network Interface” block 2016, and is coupled to correspondinginterface devices in other data processing systems via communicationnetwork interface 2024. Data processing system 2000 can include, forexample, one or more computers, such as a personal computer,workstation, mainframe, laptop, and the like.

The user interface input devices 2018 are not limited to any particulardevice, and can typically include, for example, a keyboard, pointingdevice, mouse, scanner, interactive displays, touchpad, joysticks, etc.Similarly, various user interface output devices can be employed in asystem of the invention, and can include, for example, one or more of aprinter, display (e.g., visual, non-visual) system/subsystem,controller, projection device, audio output, and the like.

Storage subsystem 2006 maintains the basic required programming,including computer readable media having instructions (e.g., operatinginstructions, etc.), and data constructs. The program modules discussedherein are typically stored in storage subsystem 2006. Storage subsystem2006 typically includes memory subsystem 2008 and file storage subsystem2014. Memory subsystem 2008 typically includes a number of memories(e.g., RAM 2010, ROM 2012, etc.) including computer readable memory forstorage of fixed instructions, instructions and data during programexecution, basic input/output system, etc. File storage subsystem 2014provides persistent (non-volatile) storage for program and data files,and can include one or more removable or fixed drives or media, harddisk, floppy disk, CD-ROM, DVD, optical drives, and the like. One ormore of the storage systems, drives, etc., may be located at a remotelocation, such coupled via a server on a network or via theinternet/World Wide Web. In this context, the term “bus subsystem” isused generically so as to include any mechanism for letting the variouscomponents and subsystems communicate with each other as intended andcan include a variety of suitable components/systems that would be knownor recognized as suitable for use therein. It will be recognized thatvarious components of the system can be, but need not necessarily be atthe same physical location, but could be connected via variouslocal-area or wide-area network media, transmission systems, etc.

Scanner 2020 includes any means for obtaining a digital representation(e.g., images, surface topography data, etc.) of a patient's teeth(e.g., by scanning physical models of the teeth such as casts 2027, byscanning impressions taken of the teeth, or by directly scanning theintraoral cavity), which can be obtained either from the patient or fromtreating professional, such as an orthodontist, and includes means ofproviding the digital representation to data processing system 2000 forfurther processing. Scanner 2020 may be located at a location remotewith respect to other components of the system and can communicate imagedata and/or information to data processing system 2000, for example, viaa network interface 2024. Fabrication machine 2022 fabricates appliances2023 based on a treatment plan, including data set information receivedfrom data processing system 2000. Fabrication machine 2022 can, forexample, be located at a remote location and receive data setinformation from data processing system 2000 via network interface 2024.The camera 2025 may include any image capture device configured tocapture still images or movies. The camera 2025 may facilitate capturingvarious perspectives of a patient's dentition. In some implementations,the camera 2025 may facilitate capture of images at various focallengths and distances from the patient.

The data processing aspects of the methods described herein can beimplemented in digital electronic circuitry, or in computer hardware,firmware, software, or suitable combinations thereof. Data processingapparatus can be implemented in a computer program product tangiblyembodied in a machine-readable storage device for execution by aprogrammable processor. Data processing blocks can be performed by aprogrammable processor executing program instructions to performfunctions by operating on input data and generating output. The dataprocessing aspects can be implemented in one or more computer programsthat are executable on a programmable system, the system including oneor more programmable processors operably coupled to a data storagesystem. Generally, a processor will receive instructions and data from aread-only memory and/or a random access memory. Storage devices suitablefor tangibly embodying computer program instructions and data includeall forms of nonvolatile memory, such as: semiconductor memory devices,such as EPROM, EEPROM, and flash memory devices; magnetic disks such asinternal hard disks and removable disks; magneto-optical disks; andCD-ROM disks.

Although the detailed description contains many specifics, these shouldnot be construed as limiting the scope of the disclosure but merely asillustrating different examples and aspects of the present disclosure.It should be appreciated that the scope of the disclosure includes otherembodiments not discussed in detail above. Various other modifications,changes and variations which will be apparent to those skilled in theart may be made in the arrangement, operation and details of themethods, systems, and apparatus of the present disclosure providedherein without departing from the spirit and scope of the invention asdescribed herein.

As used herein the terms “dental appliance,” and “tooth receivingappliance” are treated synonymously. As used herein, a “dentalpositioning appliance” or an “orthodontic appliance” may be treatedsynonymously, and may include any dental appliance configured to changethe position of a patient's teeth in accordance with a plan, such as anorthodontic treatment plan. A “patient,” as used herein may include anyperson, including a person seeking dental/orthodontic treatment, aperson undergoing dental/orthodontic treatment, and a person who haspreviously undergone dental/orthodontic treatment. A “patient” mayinclude a customer or a prospective customer of orthodontic treatment,such as a person who is using the visualization tools herein to inform adecision to undergo orthodontic treatment at all or a decision to selecta specific orthodontic treatment plan. A “dental positioning appliance”or “orthodontic appliance,” as used herein, may include a set of dentalappliances configured to incrementally change the position of apatient's teeth over time. As noted herein, dental positioningappliances and/or orthodontic appliances may comprise polymericappliances configured to move a patient's teeth in accordance with anorthodontic treatment plan.

As used herein the term “and/or” may be used as a functional word toindicate that two words or expressions are to be taken together orindividually. For example, the phrase “A and/or B” encompasses A alone,B alone, and A and B together. Depending on context, the term “or” neednot exclude one of a plurality of words/expressions. As an example, thephrase “A or B” need not exclude A and B together.

As used herein the terms “torque” and “moment” are treated synonymously.

As used herein a “moment” may encompass a force acting on an object suchas a tooth at a distance from a center of resistance. The moment may becalculated with a vector cross product of a vector force applied to alocation corresponding to a displacement vector from the center ofresistance, for example. The moment may comprise a vector pointing in adirection. A moment opposing another moment may encompass one of themoment vectors oriented toward a first side of the object such as thetooth and the other moment vector oriented toward an opposite side ofthe object such as tooth, for example. Any discussion herein referringto application of forces on a patient's teeth is equally applicable toapplication of moments on the teeth, and vice-versa.

As used herein a “plurality of teeth” may encompass two or more teeth. Aplurality of teeth may, but need not, comprise adjacent teeth. In someembodiments, one or more posterior teeth comprises one or more of amolar, a premolar, or a canine, and one or more anterior teethcomprising one or more of a central incisor, a lateral incisor, acuspid, a first bicuspid, or a second bicuspid.

The embodiments disclosed herein may be well suited for moving one ormore teeth of the first group of one or more teeth or moving one or moreof the second group of one or more teeth, and combinations thereof.

The embodiments disclosed herein may be well suited for combination withone or more commercially available tooth moving components such asattachments and polymeric shell appliances. In some embodiments, theappliance and one or more attachments are configured to move one or moreteeth along a tooth movement vector comprising six degrees of freedom,in which three degrees of freedom are rotational and three degrees offreedom are translation.

Repositioning of teeth may be accomplished with the use of a series ofremovable elastic positioning appliances such as the Invisalign® systemavailable from Align Technology, Inc., the assignee of the presentdisclosure. Such appliances may have a thin shell of elastic materialthat generally conforms to a patient's teeth but is slightly out ofalignment with an initial or immediately prior tooth configuration.Placement of the appliance over the teeth applies controlled forces inspecific locations to gradually move the teeth into the newconfiguration. Repetition of this process with successive appliancescomprising new configurations eventually moves the teeth through aseries of intermediate configurations or alignment patterns to a finaldesired configuration. Repositioning of teeth may be accomplishedthrough other series of removable orthodontic and/or dental appliances,including polymeric shell appliances.

A computer system, as used in this paper, is intended to be construedbroadly. In general, a computer system will include a processor, memory,non-volatile storage, and an interface. A typical computer system willusually include at least a processor, memory, and a device (e.g., a bus)coupling the memory to the processor. The processor can be, for example,a general-purpose central processing unit (CPU), such as amicroprocessor, or a special-purpose processor, such as amicrocontroller.

The memory can include, by way of example but not limitation, randomaccess memory (RAM), such as dynamic RAM (DRAM) and static RAM (SRAM).The memory can be local, remote, or distributed. The bus can also couplethe processor to non-volatile storage. The non-volatile storage is oftena magnetic floppy or hard disk, a magnetic-optical disk, an opticaldisk, a read-only memory (ROM), such as a CD-ROM, EPROM, or EEPROM, amagnetic or optical card, or another form of storage for large amountsof data. Some of this data is often written, by a direct memory accessprocess, into memory during execution of software on the computersystem. The non-volatile storage can be local, remote, or distributed.The non-volatile storage is optional because systems can be created withall applicable data available in memory.

Software is typically stored in the non-volatile storage. Indeed, forlarge programs, it may not even be possible to store the entire programin the memory. Nevertheless, it should be understood that for softwareto run, if necessary, it is moved to a computer-readable locationappropriate for processing, and for illustrative purposes, that locationis referred to as the memory in this paper. Even when software is movedto the memory for execution, the processor will typically make use ofhardware registers to store values associated with the software, andlocal cache that, ideally, serves to speed up execution. As used herein,a software program is assumed to be stored at an applicable known orconvenient location (from non-volatile storage to hardware registers)when the software program is referred to as “implemented in acomputer-readable storage medium.” A processor is considered to be“configured to execute a program” when at least one value associatedwith the program is stored in a register readable by the processor.

In one example of operation, a computer system can be controlled byoperating system software, which is a software program that includes afile management system, such as a disk operating system. One example ofoperating system software with associated file management systemsoftware is the family of operating systems known as Windows® fromMicrosoft Corporation of Redmond, Wash., and their associated filemanagement systems. Another example of operating system software withits associated file management system software is the Linux operatingsystem and its associated file management system. The file managementsystem is typically stored in the non-volatile storage and causes theprocessor to execute the various acts required by the operating systemto input and output data and to store data in the memory, includingstoring files on the non-volatile storage.

The bus can also couple the processor to the interface. The interfacecan include one or more input and/or output (I/O) devices. Dependingupon implementation-specific or other considerations, the I/O devicescan include, by way of example but not limitation, a keyboard, a mouseor other pointing device, disk drives, printers, a scanner, and otherI/O devices, including a display device. The display device can include,by way of example but not limitation, a cathode ray tube (CRT), liquidcrystal display (LCD), or some other applicable known or convenientdisplay device. The interface can include one or more of a modem ornetwork interface. It will be appreciated that a modem or networkinterface can be considered to be part of the computer system. Theinterface can include an analog modem, ISDN modem, cable modem, tokenring interface, satellite transmission interface (e.g. “direct PC”), orother interfaces for coupling a computer system to other computersystems. Interfaces enable computer systems and other devices to becoupled together in a network.

The computer systems can be compatible with or implemented as part of orthrough a cloud-based computing system. As used in this paper, acloud-based computing system is a system that provides virtualizedcomputing resources, software and/or information to end user devices.The computing resources, software and/or information can be virtualizedby maintaining centralized services and resources that the edge devicescan access over a communication interface, such as a network. “Cloud”may be a marketing term and for the purposes of this paper can includeany of the networks described herein. The cloud-based computing systemcan involve a subscription for services or use a utility pricing model.Users can access the protocols of the cloud-based computing systemthrough a web browser or other container application located on theirend user device.

A computer system can be implemented as an engine, as part of an engineor through multiple engines. As used in this paper, an engine includesone or more processors or a portion thereof. A portion of one or moreprocessors can include some portion of hardware less than all of thehardware comprising any given one or more processors, such as a subsetof registers, the portion of the processor dedicated to one or morethreads of a multi-threaded processor, a time slice during which theprocessor is wholly or partially dedicated to carrying out part of theengine's functionality, or the like. As such, a first engine and asecond engine can have one or more dedicated processors or a firstengine and a second engine can share one or more processors with oneanother or other engines. Depending upon implementation-specific orother considerations, an engine can be centralized or its functionalitydistributed. An engine can include hardware, firmware, or softwareembodied in a computer-readable medium for execution by the processor.The processor transforms data into new data using implemented datastructures and methods, such as is described with reference to the FIGS.in this paper.

The engines described in this paper, or the engines through which thesystems and devices described in this paper can be implemented, can becloud-based engines. As used in this paper, a cloud-based engine is anengine that can run applications and/or functionalities using acloud-based computing system. All or portions of the applications and/orfunctionalities can be distributed across multiple computing devices,and need not be restricted to only one computing device. In someembodiments, the cloud-based engines can execute functionalities and/ormodules that end users access through a web browser or containerapplication without having the functionalities and/or modules installedlocally on the end-users' computing devices.

As used in this paper, datastores are intended to include repositorieshaving any applicable organization of data, including tables,comma-separated values (CSV) files, traditional databases (e.g., SQL),or other applicable known or convenient organizational formats.Datastores can be implemented, for example, as software embodied in aphysical computer-readable medium on a specific-purpose machine, infirmware, in hardware, in a combination thereof, or in an applicableknown or convenient device or system. Datastore-associated components,such as database interfaces, can be considered “part of a datastore,part of some other system component, or a combination thereof, thoughthe physical location and other characteristics of datastore-associatedcomponents is not critical for an understanding of the techniquesdescribed in this paper.

Datastores can include data structures. As used in this paper, a datastructure is associated with a particular way of storing and organizingdata in a computer so that it can be used efficiently within a givencontext. Data structures are generally based on the ability of acomputer to fetch and store data at any place in its memory, specifiedby an address, a bit string that can be itself stored in memory andmanipulated by the program. Thus, some data structures are based oncomputing the addresses of data items with arithmetic operations; whileother data structures are based on storing addresses of data itemswithin the structure itself. Many data structures use both principles,sometimes combined in non-trivial ways. The implementation of a datastructure usually entails writing a set of procedures that create andmanipulate instances of that structure. The datastores, described inthis paper, can be cloud-based datastores. A cloud based datastore is adatastore that is compatible with cloud-based computing systems andengines.

Although reference is made to an appliance comprising a polymeric shellappliance, the embodiments disclosed herein are well suited for use withmany appliances that receive teeth, for example appliances without oneor more of polymers or shells. The appliance can be fabricated with oneor more of many materials such as metal, glass, reinforced fibers,carbon fiber, composites, reinforced composites, aluminum, biologicalmaterials, and combinations thereof for example. The appliance can beshaped in many ways, such as with thermoforming or direct fabrication asdescribed herein, for example. Alternatively or in combination, theappliance can be fabricated with machining such as an appliancefabricated from a block of material with computer numeric controlmachining. Additionally, though reference is made herein to orthodonticappliances, at least some of the techniques described herein may applyto restorative and/or other dental appliances, including withoutlimitation crowns, veneers, teeth-whitening appliances, teeth-protectiveappliances, etc.

While preferred embodiments of the present invention have been shown anddescribed herein, it will be obvious to those skilled in the art thatsuch embodiments are provided by way of example only. Numerousvariations, changes, and substitutions will now occur to those skilledin the art without departing from the invention. It should be understoodthat various alternatives to the embodiments of the invention describedherein may be employed in practicing the invention. It is intended thatthe following claims define the scope of the invention and that methodsand structures within the scope of these claims and their equivalents becovered thereby.

1. A computer-implemented method of simulating orthodontic treatment,the computer-implemented method comprising: capturing a first 2D image,the first 2D image comprising a representation of a patient's face and apatient's teeth; identifying one or more shapes associated with at leastone of the patient's teeth; building a parametric 3D model of thepatient's teeth based on the first 2D image, using one or morecase-specific parameters for the one or more shapes associated with theat least one of the patient's teeth; simulating a position of thepatient's teeth by rendering the parametric 3D model with the patient'steeth in a predetermined position; simulating a patient's gingiva bybuilding a parameterized gingiva line comprising patient-specific inputparameters; and rendering a second 2D image representing the patient'sface, the second 2D image representing the patient's teeth in accordancewith the predetermined position and the patient's gingiva in accordancewith the parameterized gingiva line.
 2. The computer-implemented methodof claim 1, wherein building the parameterized gingiva line comprises:determining one or more edges of teeth, one or more of edges of gingiva,and one or more edges of lips in the first 2D image; determining thepatient-specific input parameters from the one or more edges of teeth,the one or more of edges of gingiva, and the one or more edges of lips.3. The computer-implemented method of claim 2, wherein building theparameterized gingiva line further comprises modifying thepatient-specific input parameters based on simulated input parameters.4. The computer-implemented method of claim 1, wherein thepatient-specific input parameters comprise one or more of: a maximaldistance between a lip and a gingiva line, a minimal distance betweenthe lip and the gingiva line, or a width of a tooth at an incisal edge.5. The computer-implemented method of claim 3, wherein the simulatedinput parameters comprise one or more of: a gingiva tip parameter, anideal tooth height, a tooth thickness, a distance from the gingiva lineto the lip, or a visible tooth height.
 6. The computer-implementedmethod of claim 3, wherein building the parametrized gingiva linecomprises: modeling the parametric 3D model of the patient's teeth as acylinder; identifying coordinates on the cylinder defined by thesimulated input parameters comprising, a gingiva tip parameter, avisible tooth height, and a distance from a gingiva line to the lip;cutting the cylinder cross-sectionally by a tilted plane; and simulatingthe parameterized gingiva line as the line of intersection between thetilted plane and the cylinder projected onto the parametric 3D model. 7.The computer-implemented method of claim 1, further comprising renderinga leveled gum line, wherein rendering the leveled gum line comprisesaligning coordinates of the parameterized gingiva line.
 8. Thecomputer-implemented method of claim 7, wherein the coordinates of theparameterized gingiva line are aligned to a second degree polynomial. 9.The computer-implemented method of claim 2, further comprising: findingedges of teeth, gingiva, and lips in the first 2D image; and aligningthe parametric 3D tooth model to the edges of the teeth, gingiva, andlips in the first 2D image.
 10. The computer-implemented method of claim1, wherein the first 2D image comprises a profile image representing aprofile of the patient's face.
 11. The computer-implemented method ofclaim 1, wherein the predetermined position comprises an initialposition.
 12. The computer-implemented method of claim 1, wherein thepredetermined position comprises a final position.
 13. Thecomputer-implemented method of claim 12, wherein the final positioncomprises a position after an orthodontic treatment plan, a restorativetreatment plan, or some combination thereof.
 14. Thecomputer-implemented method of claim 1, wherein capturing the first 2Dimage comprises: instructing a mobile phone or a camera to image thepatient's face, or gathering the first 2D image from a storage device ora networked system.
 15. The computer-implemented method of claim 1,wherein building the parametric 3D model of the patient's teeth based onthe first 2D image, using the one or more case-specific parameters forthe one or more shapes associated with the at least one of the patient'steeth comprises: coarsely aligning teeth represented in the parametric3D model to the patient's teeth represented in the first 2D image; andexecuting an expectation step to determine a probability that aprojection of a silhouette of the parametric 3D model matches one ormore edges of the first 2D image a first time.
 16. Thecomputer-implemented method of claim 15, wherein building the parametric3D model of the patient's teeth based on the first 2D image, using theone or more case-specific parameters for the one or more shapesassociated with the at least one of the patient's teeth comprises:executing a maximization step using a small angle approximation tolinearize the rigid transformation of the teeth in the parametric 3Dmodel; and executing the expectation step to determine a probabilitythat a projection of a silhouette of the parametric 3D model matches theedges of the 2D image a second time.
 17. The computer-implemented methodof claim 15, further comprising: iterating though the expectation andmaximization steps a first plurality of times with a first subset ofparameters; and after iterating though the expectation and maximizationsteps the first plurality of times with the first subset of parametersof the parametric 3D model, iterating though the expectation andmaximization steps a second plurality of times with the first subset ofparameters and a second subset of parameters.
 18. A computer-implementedmethod of simulating a patient's gingiva, the computer-implementedmethod comprising: capturing a first 2D image of a patient's face,including their teeth, lips, and gingiva; building a parametric model ofthe patient's gingiva based on the first 2D image, the parametric modelcomprising patient-specific input parameters for the patient's gingiva;rendering a second 2D image of the patient's face with teeth and gingivabased on the parametric model of the patient's gingiva.
 19. Thecomputer-implemented method of claim 18, wherein building the parametricmodel of the patient's gingiva comprises: determining edges of teeth,gingiva, and lips in the first 2D image; determining thepatient-specific input parameters based on the determined edges of theteeth, the lips, and the gingiva.
 20. The computer-implemented method ofclaim 18, wherein the computer-implemented method further comprisesbuilding a parametric 3D model of the patient's teeth based on the first2D image, using one or more case-specific parameters for one or moreshapes associated with at least one of the patient's teeth.
 21. Thecomputer-implemented method of claim 20, wherein building the parametricmodel of the patient's gingiva further comprises modifying thepatient-specific input parameters based on a plurality of simulatedinput parameters, wherein one or more of the plurality of simulatedinput parameters are based on the parametric 3D model of the patient'steeth.
 22. The computer-implemented method of claim 18, wherein thepatient-specific input parameters comprise one or more of a maximaldistance between the lip and a gingiva line, a minimal distance betweenthe lip and the gingiva line, and a width of a tooth at an incisal edge.23. The computer-implemented method of claim 21, wherein the pluralityof simulated input parameters comprise one or more of: a gingiva tipparameter, an ideal tooth height, a tooth thickness, a distance from agingiva line to the lip, or a visible tooth height.
 24. Thecomputer-implemented method of claim 21, wherein building the parametricmodel of the patient's gingiva comprises: modeling the parametric 3Dmodel of the patient's teeth as a cylinder; identifying coordinates on acylinder defined by the plurality of simulated input parameterscomprising, a gingiva tip parameter, a visible tooth height, and adistance from a gingiva line to the lip; cutting the cylindercross-sectionally by a tilted plane; and simulating a parameterizedgingiva line as the line of intersection between the tilted plane andthe cylinder.
 25. The computer-implemented method of claim 18, furthercomprising rendering a leveled gum line, wherein rendering the leveledgum line comprises aligning coordinates of the parametric model of thepatient's gingiva.
 26. The computer-implemented method of claim 25,wherein the coordinates of the parametric model of the patient's gingivaare aligned to a second degree polynomial.
 27. The computer-implementedmethod of claim 20, further comprising: finding edges of teeth, gingiva,and lips in the first 2D image; and aligning the parametric model to theedges of the teeth, the gingiva, and the lips in the first 2D image. 28.The computer-implemented method of claim 18, wherein the first 2D imagecomprises a profile image representing a profile of the patient's face.29. The computer-implemented method of claim 19, further comprisingstoring the patient-specific parameters of the parametric model of thepatient's gingiva that align the parametric model of the patient'sgingiva with the edges of the teeth, the gingiva, and the lips in thefirst 2D image. 30-37. (canceled)
 38. The computer-implemented method ofclaim 18, further comprising rendering a leveled gum line, whereinrendering the leveled gum line comprises aligning coordinates of theparametric model of the patient's gingiva. 39-42. (canceled)
 43. Asystem comprising: a photo parameterization engine configured togenerate a parameterized gingiva line and a 3D parametric arch modelfrom a 2D image of a patient's face, gingiva, and teeth, the 3Dparametric arch model including case-specific parameters for a shape ofat least one of the patient's teeth, the parameterized gingiva lineincluding patient-specific input parameters for a shape of the gingiva;and a parametric treatment prediction engine configured to simulateorthodontic treatment of a patient based on the 3D parametric arch modeland historic models of a plurality of patients. 44-45. (canceled)